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SECTION  1 
INTRODUCTION 


1.1  GENERAL 

This  shjdy,  entitled  "Development  of  a Quality  Assurance  Methodology  for  the  Tech- 
nical Information  Generation  Subsystem  of  NTIPS",  has  been  prepared  by  Hagemon, 
Incorporated  (HI),  of  Fort  Worth,  Texas,  in  accordance  with  subcontract  No.  7040 
to  Operations  Research  Incorporated  of  Silver  Springs,  Maryland,  under  prime  contract 
N0019-78-C-0175.  This  study  is  designed  to  address  the  problems  and  requirements 
of  o Quality  Assurance  (QA)  program  as  part  of  the  overall  Navy  Technical  Information 
Presentation  System  (NTIPS)  development  effort.  The  following  paragraphs  describe 
the  study  methodology,  and  present  a description  of  the  current  technical  manual  and 
QA  programs,  as  well  as  a functional  description  of  NTIPS.  The  study  itself  focuses 
on  the  Technical  Information  (Tl)  Generation  phase  of  NTIPS  and  the  development  of 
a QA  system  supporting  that  phase.  Quality  Assurance,  for  the  purposes  of  this  study, 
is  defined  as  the  planned  systematic  activities  and  actions  necessary  to  provide  adequate 
confidence  that  the  item  or  product  conforms  to  established  technical  requirements.  A 
QA  philosophy  Is  developed  and  requirements  are  identified.  Finally,  a procurement 
specification  for  a QA  program,  supporting  Data  Item  Descriptions  (DID's),  and  a sample 
Contract  Data  Requirement  List  (CDRL)  were  prepared. 

1.2  APPROACH 

The  approach  used  in  the  study  included  a review  of  current  Navy  Instructions,  Technical 
Manual  and  QA  Specifications,  and  DID's,  In  addition,  research  reports  in  the  area  of 
readability  and  comprehensibility  and  NTIPS  were  reviewed.  Both  Integrated  Logistics 
Support  (ILS)  and  Instructional  System  Development  (ISD)  documentation  were  included 
in  the  review.  Following  the  review  of  the  documentation,  listed  in  Appendix  "B"  of 
this  report,  a functional  description  of  both  current  technical  publications  procurement 
and  of  NTIPS  was  prepared.  An  attempt  was  made  to  include  all  essential  processes. 

The  processes  were  then  analyzed  to  determine  feasibility  of  QA  program  monitoring. 
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A QA  program  philosophy  was  synthesized.  Within  the  framework  of  that  philosophy, 
NTIPS  processes  were  analyzed  and  QA  requirements  developed.  Based  on  the  QA 
requirements,  formal  procurement  documentation  was  prepared  in  the  form  of  a specifi- 
cation and  DID's.  The  procurement  documentation  was  further  supported  by  including 
a sample  CDRL  as  an  example  of  the  utilization  of  the  DID's. 

1.3  SCOPE  OF  CURRENT  STUDY 

It  was  determined  in  an  earlier  H!  study,  "The  Potential  Impact  of  an  Effective  Quality 

Assurance  Program  for  Navy  Technical  Publications  on  Navy  Training  and  Training 

1 * 

Equipment  Procurement",  that  further  study  should  be  directed  to  specific  areas 
of  NTIPS.  The  study  indicated  that  the  QA  process  from  the  replication  phase  through 
the  verification  phase  is  reasonably  well  covered  by  existing  procurement  documentation, 
but  that  procurement  documentation  for  the  QA  process  prior  to  replication  is  essentially 
non-existent.  The  area  in  which  few  QA  dollars  and  little  QA  effort  is  expended  occurs 
during  the  Tl  generation  phase.  It  appears  that  approximately  twenty  percent  (20%)  of 
the  QA  effort  is  expended  prior  to  replication  and  eighty  percent  (80%)  following  the 
initiation  of  replication.  The  contractor,  on  the  other  hand,  has  already  expended  on 
the  system  at  least  eighty  percent  (80%)  of  his  effort  prior  to  replication  and  only  twenty 
percent  (20%)  following  the  initiation  of  replication.  Thus  it  would  appear  that  twenty 
percent  (20%)  of  the  QA  effort  is  now  applied  to  eighty  percent  (80%)  of  the  contractor 
Technical  Publications  effort.  This  disparity  in  resource  utilization  may  be  a contributing 
factor  to  current  Tl  problems.  Thus,  this  study  program  is  directed  toward  those  activities 
which  occur  prior  to  replication  in  the  NTIPS  Tl  Definition  and  Tl  Generation  phases. 

The  Tl  Definition  phase  is  primarily  a Navy  responsibility  to  which  QA  requirements  are 
not  directed.  The  Tl  Generation  phase  is  primarily  a contractor  responsibility  to  which 
a QA  program  is  applicable  and  essential.  Thus,  it  is  appropriate  thot  a major  effort  in 
definitizing  an  effective  QA  program  be  constrained  to  the  Tl  Generation  phase. 


Note;  References  to  corresponding  numbers  are  given  in  the  Bibliography,  following 
Section  5. 
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However,  !n  order  to  understand  all  of  the  Implicot'ons  of  the  processes  which  occur 
during  the  Tl  Generation  phase,  it  was  necessary  ro  review  and  define  as  precisely  os 
possible  the  NTIPS  process.  Accordingly,  the  initial  parts  of  the  report  address  NTIPS 
in  order  to  identify  NTIPS  functional  processes.  The  development  of  QA  requirements, 
however,  was  limited  to  the  Tl  Generation  phases  of  the  program. 

1.4  CONTENT  OF  THE  REPORT 

Section  2 of  the  report  provides  a description  of  a Technical  Publications  development 
process  Quality  Assurance  program  which  is  representative  of  current  programs,  followed 
by  a brief  description  of  NTIPS.  Section  3 of  the  report  contains  a drtailed  discussion 
of  the  Tl  Generation  phase  of  NTIPS.  Assumptions  made  in  generating  greater  detail 
are  provided  in  order  to  support  the  discussion.  Section  4 of  the  report  describes  the 
QA  requirements  derived  from  the  detailed  analysis  of  the  Tl  Generation  phase  of 
NTIPS  described  in  Section  3.  Section  5 of  the  report  is  a brief  presentation  of  the 
results  and  conclusions  of  the  study.  In  addition  to  the  Bibliography,  the  report  also 
contains  two  Appendices:  1)  Appendix  "A",  Glossary  of  Terms,  and  2)  Appendix  "B", 
Review  Documents  Matrices. 


SECTION  2 

DESCRIPTION  OF  PROCESSES 

2.1  CURRENT  PROCESSES 

The  current  Technical  Information  (Tl)  development  processes  and  the  QA  of  the  Tl 
will  vary  slightly  from  one  contractor  to  another.  Therefore,  the  discussion  and  se- 
quence of  the  Tl  production  process  presented  here  are  considered  typical  of,  but  not 
necessarily  applicable  to,  any  particular  contractor.  The  process,  as  presented,  is 
the  result  of  reviewing  several  Tl  production  processes  documented  in  the  literature. 

2.1.1  TECHNICAL  PUBLICATIONS  DEVELOPMENT 

A diagram  depicting  a typical  current  Tl  development  process  is  given  in  Figure  2-1. 

A description  of  the  process  by  the  block  numbers  follows.  This  section  does  not  re- 
flect recommended  NTIPS  procedures. 

Blocks  1 and  2 describe  the  input  information  the  writer  will  use  to  prepare  the  tech- 
nical information.  Block  1 describes  the  "in-house"  input  or  the  data  available  at 
the  contractor  facility.  These  include  items  such  as  engineering  design  drawings, 
maintainability  data,  safety  or  hazard  analyses,  vendor  brochures  and  data,  support 
equipment  specifications,  task  analyses,  and  similar  informatian.  This  block  will  also 
include  information  obtainable  from  conferences  with  design,  maintainability,  manu- 
facturing, training,  safety,  or  other  personnel.  The  customer  data  input  represented 
in  Block  2 are  Items  such  as  contractural  specifications  for  the  technical  information, 
manual  specifications,  equipment  unsatisfactory  reports  (UR),  reports  on  Government 
Furnished  Equipment  (GFE)  items,  and  other  data  that  the  customer  supplies  relative 
to  the  Tl , 

In  Block  3,  the  writer  prepares  an  outline  of  the  Tl  to  be  presented  from  the  information 
supplied  in  Blocks  1 and  2.  From  the  outline  the  writer  determines  requirements  for  text 
and  illustrations  as  represented  in  Block  4.  Some  judgment  is  required  by  the  writer  to 


2-1 


ascertain  the  level  of  detail  to  be  presented  in  the  text  and  illustrations,  and  the  deci- 
sions are  generally  based  on  past  experience. 

Block  5 indicates  the  completion  of  a draft  of  the  text.  The  illustrators  then  start 
preparation  of  the  art  work  (as  shown  by  Block  6)  and  the  Engineering  Department  re- 
views the  draft  of  the  text  as  represented  by  Block  7.  The  Engineering  review  is  a 
technical  review  of  the  content  of  the  text  draft. 

The  illustrators'  woHt  results  in  preliminary  art  work  as  cited  in  Block  8 which  later 
results  in  the  art  work  for  the  manuscript. 

Comments,  deficiencies,  and  discrepancies  resulting  from  the  engineering  review  are 
incorporated  as  corrections  to  the  draft  text  in  Block  9.  The  corrected  or  revised  text 
is  then  combined  with  the  preliminary  art  work  to  produce  the  preliminary  Tl  copy  in 
Block  10. 

The  maintenance  procedures  as  described  in  the  Tl  are  then  validated  in  Block  11. 

In  the  validation  process,  the  Tl  is  used,  usually  by  the  writers,  to  perform  the  specific 
task  or  series  of  tasks  on  the  specified  hardware.  This,  in  essence,  is  a functional  test 
of  the  Tl  performed  by  the  contractor  technical  writer. 

The  Block  12  art  revision  and  Block  13  text  revision  requirements  may  result  from  the 
validation  of  the  Tl.  After  required  text  revisions  are  made  in  Block  13,  the  text  is 
edited  in  Block  14.  The  editing  is  generally  performed  by  the  more  experienced  tech- 
nical writers  with  emphasis  on  the  grammatical  structure  of  the  text.  If  the  validation 
process  uncovers  no  faults  in  the  Tl,  then  these  revision  requirements  drop  out. 

After  editing,  the  reproducible  typographic  composition  of  the  text  is  carried  out  in 
Block  15.  Block  16  describes  the  task  of  proofing  the  text  which  is  performed  by  the 
writer  and/or  editora.  During  preparation  of  the  text,  the  art  work  is  being  readied 
for  reproduction  as  shown  in  Block  17, 
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In  the  task  represented  by  Block  18,  the  final  art  work  and  proofed  text  are  assembled 
into  a reproducible  master.  At  this  stage,  comprehensive  production  quality  checks, 
represented  by  Block  19  are  made.  These  checks  generally  include  title  page  and 
front  matter  format,  text  completeness  and  continutity,  figure  references  and  titles, 
proper  page  numbering,  classification  and  specification  compatibility.  The  Block  20 
corrections  are  made  to  the  manuscript  as  required.  Verification  activity  represented 
in  Block  21  is  a final  test  of  the  manual  as  to  its  applicability  to  the  hardware  and  task 
for  which  the  Tl  was  written.  Verification  is  a formal  process  through  which  users  (Navy 
personnel)  determine  the  adequacy  and  accuracy  of  the  technical  data  by  actual  per- 
formance of  the  procedures  defined  by  the  information.  The  user  review  results  in 
identifying  omissions  and  correcting  deficiencies  such  as  inaccurate  procedures,  or 
inaccurate  part  numbers,  tolerances,  torque  values,  instrument  settings  and  readings, 
found  in  the  Tl.  These  required  corrections  are  represented  by  Block  22. 

Copies  for  Navy  review  are  produced  as  designated  in  Block  23  and  the  review  package 
is  sent  to  the  customer  as  denoted  in  Block  24.  At  the  completion  of  the  review,  the 
customer  either  provides  comments.  Block  25,  or  approves  the  review  copy.  Block  26. 

If  the  customer  has  made  comments  for  incorporation  into  the  Tl,  these  are  incorporated 
in  Block  27  and  the  copy  is  again  sent  to  the  customer  for  review  and  approval  as  indi- 
cated by  Block  28, 

Following  customer  approval,  the  Tl  package  is  released  for  production.  The  specified 
production  process  depends  upon  the  medium  or  media  of  the  Tl.  Block  29  represents 
the  release  for  production  and  Block  30  represents  the  distribution  of  the  Tl  to  the 
different  using  activities, 

2.1.2  QUALITY  ASSURANCE  (QA) 

Currently,  Quality  Assurance  (QA)  of  Tl  is  minimal  within  the  typical  production  process. 
Only  three  formal  inspections  are  performed  during  the  process.  These  are  validation, 
production  quality  checks,  and  verification;  and  are  represented  in  Figure  2-1  by 
Blocks  11,  19,  and  21  respectively. 


The  validation  procedure  is  classified  as  formal  because  it  is  a contractually  required 
procedure  that  ensures  that  the  Tl  accurately  supports  the  weapon  system  and  equipment 
being  documented.  The  contractor  is  responsible  for  all  validation  procedures  and  the 
incorporation  of  corrections  for  all  technical  and  other  discrepancies  recorded  during 
validation.  Though  this  is  a contractually  required  procedure  and  aids  in  assuring 
the  quality  of  the  Tl,  the  procedure  does  not  necessarily  involve  QA  personnel,  per  se, 
of  either  the  contractor  or  the  customer. 

The  production  quality  checks  include  all  the  composition  and  format  details  of  the  Tl. 
These  checks  are  usually  performed  by  more  experienced  writers  within  the  technical 
writing  group.  They  may  be  designated  as  QA  personnel  within  the  group,  but  usually 
report  to  the  same  management  as  the  writers.  The  checks  are  generally  thorough  and 
documented,  thus  providing  a degree  of  formality. 

Verification  is  another  formal  process  in  which  the  Navy  personnel  determine  the  ade- 
quocy  and  accuracy  of  the  Tl  by  performance  of  the  procedures  using  the  Tl.  This  is 
the  customer's  check  to  assure  the  functional  adequacy  and  accuracy  of  the  data. 
Usually  only  customer  QA  personnel  become  involved  in  this  process. 

The  customer  reviews  of  Blocks  25,  26  and  28  are  not  classified  as  strictly  formal  In- 
spections because  the  amount  of  inspection  performed  here  is  minimal  in  most  instances. 
These  reviews  are  often  only  a formality  for  final  acceptance  of  the  Tl.  When  inspec- 
tions are  performed,  they  are  similar  in  nature  to  those  performed  by  the  contractor  in 
Block  19. 

Several  less  formal  reviews  and  inspections  are  performed  during  the  production  of  the 
Tl  that  aid  in  assuring  the  quality  of  the  information.  However,  these  are  usually  not 
contractural  requirements  and  are  generally  not  documented  by  formal  QA  recofUs, 
either  those  of  the  contractor  or  those  of  the  customer.  These  informal  reviews  and  in- 
spections include  engineering  reviews  (Block  7),  editing  (Block  14),  text  proofing 
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(Block  16),  and  the  checking  of  the  art  work  during  Its  preparation  (Blocks  6,  8,  and  17). 
The  engineering  review  Is  the  first  technical  review  In  the  production  of  the  Tl  and  Is  the 
only  Informal  review  or  Inspection  performed  by  personnel  outside  of  the  organization  pre- 
paring the  product.  Editing  and  text  proofing  are  performed  by  personnel  within  the  tech- 
nical writing  groups.  Art  work  Is  reviewed  and  proofed  within  the  Illustrations  group. 

In  summary,  the  contractor  performs  four  Informal  checks  and  two  formal  checks  to  assure 
the  quality  of  Tl  products.  The  customer  performs  one  formal  proofing  of  the  Information 
during  the  verification  procedure  and  one  quasl-formal  review  In  conjunction  with  the 
final  approval  of  the  material  for  production  and  distribution. 


None  of  the  QA  Inspections  of  the  contractor  are  performed  by  personnel  Independent 
of  the  technical  publications  organization.  Most  of  the  formal  checks  or  Inspections 
are  made  after  the  Tl  product  Is  essentially  complete. 


2.2  NAVY  TECHNICAL  INFORMATION  PRESENTATION  SYSTEM  (NTIPS) 

2.2.1  GENERAL 

The  objective  of  the  NTIP  Program  Is  the  development  of  a Navy-wide  system  for  the 
generation,  distribution,  control  and  update  of  Tl  required  for  operation,  maintenance, 
training,  and  logistic  support  of  hardware  systems.  NTIPS  goals  are  to  Improve  fleet 
material  and  training  readiness  by  Improving  Tl  comprehensibility  and  useablllty,  and 
by  delivery  of  Tl  prior  to  hardware  delivery;  and  to  reduce  training  time  and  life  cycle 
costs. 


The  Navy  decision  coordinating  paper  for  NTIPS  dated  8 March  1978,  indicated  that 
NTIPS  may  be  divided  into  five  major  subsystems.  These  are: 

1)  The  pre-contractual  phase  called  Technical  Information  Definition 

2)  Technical  Information  Generation  which  Is  the  Initial  contractual  phase 
and  includes  draft  preparation  and  draft  validation 
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SYMBOL  DEFINITION 

Th«  Symbol  indicofes  the  logical  AND,  which 

I means  both  (or  all)  are  necessary 

The  Symbol  Indicates  non-exclosive  OR,  which 

• mejns  one  or  both  (or  any)  may  obtain 

The  Symbol  Indicates  the  exclusive  OR,  which  means  one 

I or  the  other  but  not  both  (or  all)  may  obtain 
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3)  Technical  Information  Replication  which  describes  the  phase  during  which 
draft  material  is  printed  or  processed  in  microform  or  other  medium  in 
final  format 

4)  Distribution  and  Delivery  which  follows  the  Replication  phase 

5)  Update  and  Correction  Phase 

2.2.2  NTIPS  PROCESS 

3 

A more  recent  and  far  more  detailed  block  diagram  describing  the  NTIPS  process  is 
shown  in  Figure  2-2.  The  meaning  of  the  blocks  and  their  relation  to  the  five  NTIPS 
phases  are  discussed  in  the  following  sections  of  the  report.  This  discussion  is  intended 
to  provide  essential  and  background  information  for  the  analysis  of  the  Technical  Infor- 
mation (Tl)  Generation  phase  QA  requirements,  which,  as  described  earlier.  Is  the  central 
focus  of  this  study,  i.e.,  the  development  of  QA  requirements  for  the  Tl  Generation 
phase  of  NTIPS. 

Blocks  i through  6 of  Figure  2-2  represent  the  information  and  data  required  for  the 
initial  NTIPS  activities.  Block  7 represents  the  job  task  analysis  which  is  implemented 
to  identify  maintenance  and  operating  tasks  demanded  by  the  system.  Blocks  8 through 
1 i represent  the  products  of  that  analysis.  Block  12  represents  the  summary  of  that  analysis 
in  terms  of  the  information  needed  in  the  technical  publication.  It  is  a preliminary  def- 
inition of  the  Information  requirements  and  technical  publication  content.  Block  14, 
entitled  "The  Head/Book  Trade-Off",  has  as  inputs  Blocks  12  and  13  which  identify  the 
information  required  and  the  currently  available  training  technology.  The  Head/Book 
Trade-Off  analysis  results  in  the  information  described  in  Blocks  14.1  through  14.4  and 
15. 1 and  15.2.  The  purpose  of  the  Head/Book  Trade-Off  is  the  non  exclusive  division 
of  technical  information  requirements  for  technical  publication  and  training.  Training 
requirements  are  established  for  both  OJT  and  school  training,  as  well  as  the  description 
of  the  technical  documentation  needed  for  training  support.  Block  16,  17,  and  18  rep- 
resent the  analysis  process  for  identifying  medium  requirements,  level  of  detail  of  topic 
treatment  and  Tl  formats  as  well  as  camprehensibility  criteria  for  the  final  documenta- 
tion. Blocks  19  and  20  describe  the  task  of  developing  cost  estimates  and  preparation 
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of  RFP's  which  include  the  preparation  and/or  selection  of  Technical  Manual  Contract 
Requirements  (TMCR),  CDRL,  Specifications,  and  DID's  for  procuring  operation  and 
maintenance  documentation  for  the  equipment  of  the  system.  Block  21  represents  com- 
pletion of  those  processes  from  RFP  preparation  through  the  letting  of  the  controct  and 
includes  proposal  evaluation  and  contract  negotiations.  Block  21  represents  completion 
of  the  Tl  Definition  phase  of  NTIPS.  Since  Tl  Definition  is  a Navy  task,  QA  require- 
ments are  not  applicoble  to  any  of  the  tasks  or  activities  described  in  this  stage  of 
NTIPS.  However,  Navy  QA  personnel  must  prepare  sections  of  the  RFQ  and  verify 
the  fact  the  appropriate  QA  DID's  and  Specifications  have  been  included  in  the  bid 
package  prepared  by  the  Navy.  Thus,  while  no  formal  QA  activities  occur  during  the 
Tl  Definition  phase  there  is  a very  significant  and  essential  effort  in  preporing  for  an 
effective  QA  program  during  and  following  Tl  Generation.  This  effort  must  include  a 
requirement  for  the  contractor  in  his  proposal  to  prepare  and  submit  a formal,  detailed 
QA  Program  Plan  (QAPP).  If  this  is  not  done  by  Navy  and  contractor  in  turn  prior  to 
the  contractual  agreement  with  the  contractor,  no  effective  QA  program  will  result. 
Effective  planning  is  essential  to  the  successful  implementation  of  an  effective  QA 
program. 

The  next  series  of  blocks  describe  the  Tl  Generation  phase  which  is  of  major  interest  to 
this  study.  Blocks  22  and  23  represent  on-going  program  activities  which  include  the 
design  and  manufacture  of  hardware,  the  implementation  of  logistic  support  analysis  and 
the  preparation  of  the  logistic  support  analysis  report.  These  activities  yield  significant 
data  inputs  to  the  technical  writers'  preparation  of  the  technical  publication  draft.  In 
addition.  Blocks  24  and  25  represent  the  distribution  of  these  data  not  only  to  the  writers, 
but  also  to  the  NTIPS  data  base.  Block  26  represents  the  contractor  octivity  of  updating 
and  tailoring  of  DID's  and  Specifications  to  fit  the  specific  requirements  of  the  individual 
system  and/or  publication  product.  This  activity  is  included  to  provide  the  contractor 
with  the  opportunity  to  use  the  knowledge  gained  in  the  developing  system  to  shape  re- 
quirements to  provide  unique  and  cost  effective  solutions  to  problems.  Block  27  repre- 
sents the  activity  of  writing  and  editing  the  Tl . Block  28  represents  the  appllcotion  of 
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the  QA  to  the  Tl  Generation  phase  of  Tl  Development.  Block  29  represents  the  valida- 
tion process  in  which  the  contractor  personnel  using  a draft  of  the  technical  publication 
carry  out  the  assigned  maintenance/operation  tasks  in  order  to  determine  the  technical 
accuracy,  odequacy,  and  completeness  of  the  Tl  content.  Block  30  represents  o Navy 
activity  supported  by  the  contractor  which  is  similar  to  the  validation  process.  Verifica- 
tion is  carried  out  by  the  Navy  under  operational  conditions  to  determine  the  accuracy, 
adequacy,  and  completeness  of  the  technical  information.  It  is  frequently  required  that 
a contractor  provide  training  for  Navy  personnel  who  will  participate  in  the  verification 
process.  This  tosk  is  represented  by  Block  31.  The  verification  process  has  two  results: 

1)  an  update  and  correction  of  all  faults  found  in  the  technical  adequacy  and  accuracy 
of  the  Tl,  and  2)  to  identify  weaknesses  in  the  training  program  and  training  documenta- 
tion. These  results  ore  represented  by  Blocks  32  and  36.  Thus,  both  validation  done  by 
the  contractor,  and  verification  done  by  the  Navy  are  Tl  correction  processes.  While 
preliminary  manuals  ore  made  available  for  the  verification  process,  final  camera  ready 
or  digitized  copy  preparation  is  represented  by  Block  35  of  the  diagram.  Following  the 
preparation  of  the  master  copies  of  the  corrected  and  updated  Tl,  the  process  of  Replica- 
tion is  carried  out  as  represented  by  Block  37,  Block  38  and  39  represent  the  distribution 
of  the  replicated  copy  to  the  User  activity  or  appropriate  Shore  establishment.  Once  de- 
livered, the  process  of  error  detection  by  Users  continues.  As  errors  ore  detected  and  the 
need  for  updating  the  Tl  is  determined,  the  process  of  updating  is  initiated.  The  processes 
of  detection  and  update  ore  depicted  by  Block  42  and  44.  The  feedbock  loop  leoding  from 
Block  44  will  be  carried  as  far  back  into  theNTIPS  as  the  correction  requires  and  will  be 
determined  on  an  individual  basis. 
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SECTION  3 

DETAILED  DISCUSSION  OF 
Tl  GENERATION 

3.1  GENERAL  DISCUSSION 

In  the  previous  section,  Section  2.2,  on  upper  level  diagram  of  the  NTIPS  was  pres- 
ented and  discussed  in  order  to  provide  a background  for  a more  detailed  discussion 
of  the  Tl  Generation  phase  of  NTIPS.  The  information  presented  In  Section  2 was 
gleaned  from  reports  and  other  documents  listed  in  Appendix  B of  this  report,  as  well 
as  from  personal  contacts  with  Navy  personnel  and  industry  representatives.  Section 
3.4,  Tl  Generation  Discussion,  presents  a rather  detailed  block  diagram  and  discussion 
of  the  activities  and  processes  of  the  Tl  Generation  phase.  While  this  discussion  is 
constrained  by  the  NTIPS  requirements,  certain  assumptions  with  regard  to  NTIPS 
objectives  and  processes  are  made  which  may  not  fall  within  the  current  NTIPS  con- 
cept. The  simplified  technical  publications  paradigm.  Figure  3»1,  describes  some 
of  the  assumptions  and  objectives.  The  paradigm  shows  that  data  is  processed  accord- 
ing to  certain  rules  by  a writer  who  then  synthesizes  the  data  into  useable  information 
which  after  proper  training  is  used  by  personnel  to  perform  some  task  on  a piece  of 
equipment  which  results  in  equipment  performance.  If  the  data  are  valid;  the  rules 
are  correct;  and  the  writer  obeys  these  rules,  interprets  the  data  correctly,  and  syn- 
thesizes well;  the  assumption  is  that  useable  information  will  result.  It  is  further 
assumed  that  adequately  trained  personnel  using  good  information  can  then  perform 
tasks  on  equipment  which  will  result  in  proper  equipment  performance.  The  final 
measure  of  the  effectiveness  of  the  information  and  its  means  of  presentation  can  only 
be  the  performance  of  personnel  and  the  performance  of  the  equipment.  Direct  testing 
of  the  effectiveness  of  the  information  under  these  circumstances  is  difficult  and  further 
compounded  by  the  adequacy  of  the  training  of  the  personnel  performing  the  task. 
However,  it  moy  be  assumed  that,  if  the  data  are  incomplete  or  contain  errors,  the 
useability  of  the  information  will  be  reduced  and  task  and  equipment  performance 
im- aired.  If,  on  the  other  hand,  the  data  are  current,  complete,  and  error-free. 
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the  probability  of  obtaining  useable  information  is  enhanced  and  as  a direct  consequence 
task  and  equipment  performance  are  enhanced.  By  the  same  token,  if  the  rules  provided 
the  writer  are  clear,  easily  comprehensible,  complete,  and  correct,  the  probability  of 
obtaining  useable  information  is  enhanced.  Of  course,  the  same  thing  is  true  of  the 
writer's  skills  and  knowledge  which  are  dependent  upon  his  basic  ability  and  his  training 
If  the  writer  possesses  the  skills  and  knowledge  to  interpret  data  properly,  and  to  be 
guided  by  the  rules,  then  the  probability  of  obtaining  useable  information  is  enhanced. 
The  corollary  is  that  without  these  skills  the  writer  Is  less  likely  to  produce  useable 
information.  Briefly,  bad  data,  bad  rules,  or  bad  writers,  cannot  be  expected  to 
produce  anything  but  bad  information;  while  good  data,  good  rules,  and  good  writers, 
have  a higher  probability  of  achieving  an  acceptable  result.  The  remaining  variable 
shown  in  the  pr  rodigm  which  has  not  been  discussed  is  personnel  training.  While  tech- 
nical publications  personnel  cannot  be  expected  to  be  responsible  for  training  the 
personnel  performing  a task,  they  can  be  expected  to  prepare  documentation  designed 
specifically  for  personnel  with  a known  level  of  training  and  experience.  Once  per- 
sonnel skills  and  training  have  been  defined,  it  is  the  responsibility  of  technical  pub- 
lications writers  to  write  or  prepare  documentation  suitable  for  personnel  possessing 
those  skills. 

Tl  Generation,  unlike  Tl  Definition,  is  a contractor  process  and  responsibility,  and 
like  other  contractor  processes,  should  be  supported  by  a well-defined  Quality  Assur- 
ance program.  The  QA  program  requirements  should  be  defined  by  Procurement  Spec- 
ifications and  Data  Item  Descriptions.  The  Tl  Generation  task,  unlike  a manufacturing 
task,  is  not  iterative  in  the  same  sense  as  a machine  process.  The  development  of  each 
manual  or  each  page  of  the  manual  is  a unique  human  process  involving  both  data  in- 
terpretation and  text  synthesis.  Regardless  of  the  medium  used  in  the  presentation  of 
technical  information,  human  processes  and  activities  are  an  essential  and  crucial 
element  in  the  development  of  a technical  publication. 
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3.2  HUMAN  PROCESSES 

Figure  3-2  is  an  expanded  paradigm  of  the  technical  publication  process  and  objectives 
which  emphasizes  the  human  activities.  Writers  and  illustrators  interpret  both  the  roles 
and  the  data  and  then  proceed  to  synthesize  o product.  The  editor  presumably  interprets 
the  rules  In  the  same  manner  as  the  writer  and  illustrator  and  provides  a check  of  the  qual- 
ity of  their  performance.  Validation  is  accomplished  by  providing  maintenance  or  operator 
psrsonnel  with  a draft  of  the  publication  which  the  Operator/Maintenance  personnel  then 
use  to  perform  the  required  tasks.  The  task  of  editors  is  to  verify  the  odherence  of  the 
writer  or  illustrator  to  the  rules  which  apply  to  the  product.  The  task  of  validation  on  the 
other  hand,  does  not  test  the  draft  directly,  but  tests  whether  or  not  the  task  can  be  per- 
formed in  a satisfactory  manner  by  Operator/Maintenance  personnel.  Procedural  valida- 
tion is  therefore  the  first  test  of  the  effectiveness  of  the  manual.  It  is  possible  for  a monuol 
to  be  prepared  in  accordance  with  all  the  rules,  but  due  to  data  errors,  to  describe  task 
requirements  that  are  impossible  to  carry  out.  The  validation  process  uses  personnel  who 
are  presumably  representative  of  the  user  to  test  the  adequacy  of  the  manual  with  regard 
to  its  intended  purpose,  i.e,,  to  enable  maintenance  and  operator  penonnel  to  perform 
the  desired  tasks  successfully.  Contractor  procedural  validation  of  technical  publications 
is  an  essential  "test"  of  manual  effectiveness. 

Human  processes  are  not  as  easily  controlled  as  are  machine  processes.  A machine,  once 
set  up,  can  produce  duplicate  parts  rapidly,  efficiently,  and  accurately.  Writers  cannot 
produce  words  in  the  same  manner.  Thus,  not  only  is  it  more  difficult  to  define  the  writer's 
task,  but  it  is  also  equally  difficult  to  control  the  quality  of  his  product.  Whatever  control 
is  achieved  must  be  achieved  through  the  control  of  human  processes.  There  is  a need  to 
establish  a mechanism  which  increases  the  probability  that  the  interpretation  of  rules  and 
data  is  accurately  and  efficiently  carried  out.  There  is  a need  to  insure  that  the  synthesis 
of  text,  in  accordance  with  the  rules,  is  also  effectively  and  efficiently  implemented.  It 
is  also  essential  that  the  effectiveness  of  the  processes  be  checked  by  procedural  valida- 
tion as  part  of  the  Tl  Generation  phase  of  NTIPS.  The  contractor  must  be  made  responsible 
for  the  activities  of  Interpretation,  synthesis,  and  validation.  This  contractor  responsibility 
must  be  monitored  closely  by  Quality  Assurance  personnel  if  o quality  product  is  to  be  the 
result.  The  quality  product  in  the  case  of  NTIPS  is  effective  operator  and  maintainer  task 
performance. 
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3.3  CONTROL  OF  HUMAN  PROCESSES 

Mechanisms  for  controlling  human  behavior  are  training,  evaluation,  and  feedback. 

The  extent  and  scope  of  the  training  program  developed  by  the  contractor  for  his  tech- 
nical writers  is  crucial  to  the  quality  of  the  final  product.  Writers/I  I lustrators  should 
receive  training  in  at  least  the  following: 

• The  physical  location  of,  and  procedures  for,  acquiring  data 

• Use  of  applicable  specifications,  standards,  and  DID's 

• Utilization  of  analysis  products 

• Equipment  characteristics  and  design 

• Utilization  of  engineering  reports 

• Maintainability 

• Safety 

• Human  Engineering 

• Criticality  of  equipment  to  hardware  success 
e Use  of  Style  Guides 

e Vocabulary  limitation 

e Syntax  appropriate  to  the  needs  of  the  product 
e Availability  of  production  personnel  as  a data  resource 

Continuous  evaluation  of  the  writer's  performance  and  intermediate  products  is  essential. 

It  is  inefficient  to  wait  for  the  final  product  for  evaluation.  With  sophisticated  word- 
processing equipment  currently  available,  all  Increments,  no  matter  how  small,  of  the 
final  product  should  be  evaluated  objectively  following  initial  typing.  Objective  eval- 
uation of  the  writer's  product,  which  could  easily  be  handled  by  word-processing  equip- 
ment, include:  (1)  vocabulary  use,  (2)  sentence  length,  (3)  word  length,  and  (4)  some 
syntactical  elements.  Subjective  evaluation  by  editors  should  also  be  carried  out  concur- 
rently with  the  objective  evoluation,  but  the  editor  should  be  relieved  of  the  tedious 
tosk  associated  with  variables  which  can  be  incorporated  into  a word-processing  system 
review  of  the  text.  It  is  emphasized  that  the  writer's  first  draft  of  the  text  material 
should  be  evaluated  objectively  and  subjectively  in  such  a way  that  deficiencies  ere 
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specifically  identified  for  him.  Flesch  counts  or  other  techniques  which  yield  a score 
for  the  author's  entire  product  are  not  an  effective  means  for  controlling  writer  processes. 

Feedback  is  the  third  mechanism  for  controlling  human  behavior.  The  immediacy  of 
feedback  is  critical  to  the  effectiveness  of  that  mechanism  for  controlling  behavior. 

It  is  therefore  emphasized  that  even  the  smallest  part  of  the  writer's  product  should  re- 
ceive immediate  attention  before  It  is  returned.  When  he  turns  the  draft  into  the  typist, 
that  draft  should  be  processed  objectively  and  returned  to  him  with  the  objective  com- 
ments immediately  available.  Objective  processing  makes  possible  preliminary  editing 
of  the  text  by  an  editor,  since  a major  portion  of  the  tedious  and  time-consuming  mechan- 
ical tad<  will  be  done  for  him.  The  mechanism  for  controlling  the  writer's  performance 
then  consists  of  effective  broad-based  training,  continuous  evaluation,  and  immediate 
feedback. 

3.4  Tl  GENERATION  DISCUSSION 

Technical  Information  Generation  is  interpreted  to  include  all  essential  contractor 
activities  from  the  establishment  of  a technical  publications  contract  to  the  comple- 
tion and  submittal  of  a final  manuscript  or  final  documentation  presented  in  some  other 
medium.  There  are  six  basic  elements  or  functions  which  the  contractor  must  be  pre- 
pared to  implement.  They  are: 

(1)  Management  and  Administration 

(2)  Data  Acquisition  and  Distribution 

(3)  Analysis 

(4)  Training 

(5)  Text  Preparation 

(6)  Illustration  Preparation 

Three  of  the  above  functions  are  commonly  found  in  industry  today,  and  are  in  fact 
organizational  elements  of  the  technical  publications  organization.  These  are; 
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management,  the  writing  group,  and  the  illustrations  group.  The  other  three  functions; 
data  acquisition,  analysis,  and  training,  may  be  carried  out  implicitly  but  are  not  com- 
monly addressed  directly  through  organizational  responsibility,  planning,  or  scheduling. 


The  interrelationships  among  the  Tl  Generation  elements  and  a detailed  description  of 
essentiol  activities  is  described  in  the  block  diagram  Figure  3-3,  entitled  Technical 
Information  Generation.  In  order  to  establish  a point  of  reference  with  NTIPS,  the 
diagram  begins  with  block  number  21  entitled  Estoblish  Contract  of  Figure  2-2,  and  is 
an  expansion  of  the  information  contained  in  Figure  2-2.  All  of  the  activities  depicted 
in  the  diagram  should  be  included  explicitly  in  Tl  Generation  planning  and  scheduling. 
Moreover,  all  activities  depicted  should  be  addressed  directly  and  formally  in  technical 
publications  proposals  in  order  to  permit  evaluation  prior  to  the  establishment  of  a con- 
tract. While  the  diagram  is  functionally  oriented,  it  is  implicit  that  the  functions 
should  be  represented  by  contractor  organizational  elements.  In  the  following  discus- 
sion, when  convenient,  organizational  responsibilities  may  address  the  organizational 
element  or  group  responsibility  directly. 

3.4.1  DESCRIPTION  OF  THE  DIAGRAM 

The  Block  Diagram  of  the  Tl  Generation  phase  of  NTIPS  is  supported  by  a narrative 
explanation  which  describes  the  activity  associated  with  each  of  the  block  titles. 

These  paragraphs  are  listed  by  block  number  for  reference.  The  block  diagram  not 
only  identifies  the  octivity,  but  also  indicates  a rough  sequence  and  the  interrelation 
among  the  activities  depicted  by  the  block  titles.  The  diagram  is  read  from  left  to 
right,  and  therefore,  arrows  are  not  used. 

The  symbols  used  in  the  connecting  structure  for  the  blocks  represent  the  three  logical 
operations;  AND,  non-exclusive  OR,  and  the  exclusive  OR. 

(a)  The  Symbol  .ZK  indicates  the  logical  AND,  which  means 

both  (or  all)  ore  necessary 
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indicates  non-exclusive  OR,  w/hich  means 
one  or  both  (or  any)  may  obtain 


(b)  The  Symbol 

(c)  The  Symbol  — | indicates  the  exclusive  OR,  which  means  one 

or  the  other  but  not  both  (or  all)  may  obtain 

3.4.2  EXPLANATION  OF  THE  BLOCK  DIAGRAM 

The  following  is  a narrative  description  of  the  block  diagram  Figure  3-3  entitled 
"Technical  Information  Generation".  While  the  level  of  detail  included  in  the  descrip- 
tion below  may  be  greater  than  necessary  for  an  understanding  of  the  Tl  Generation 
process,  it  is  not  too  great  for  determining  which  processes  are  amenable  to  QA  sur- 
veillance. 

3.4.2. 1 BLOCK  NO,  1 - INITIATE  IMPLEMENTATION  OF  THE  PROGRAM  PLAN 
The  first  step  following  the  award  of  the  contract  is  to  initiate  the  implementation  of 
the  approved  Program  Plan  in  accordance  with  the  schedule  enclosed  in  that  plan. 

3. 4.2. 2 BLOCK  NO.  2 - ACTIVATE  ORGANIZATIONAL  ELEMENTS 

An  essential  first  step  In  implementing  the  Program  Plan  is  to  activate  the  oiganizational 
elements  required  by  the  plan.  It  is  presumed  that  the  first  element  to  be  activated  will 
be  Management  ond  Administration. 

3. 4. 2. 3 BLOCK  NO.  3 - MANAGEMENT  AND  ADMINISTRATION 
Management  ond  Administration  responsibilities  begin  the  first  day  following  contract 
award  and  do  not  end  until  the  final  product  contracted  for  has  been  accepted  and  de- 
livered. In  order  to  sustain  effective  management  throughout  the  development  of  Tl 
products,  responsibilities  and  authority  for  each  of  the  elements  within  the  technical 
publications  group  must  be  clearly  defined  and  specified. 

3. 4.2.4  BLOCK  NO.  3.1  - SCHEDULES  AND  PLANNING 

It  is  the  responsibility  of  the  management  and  administrative  group  to  develop  schedules 
for  both  finol  and  intermediate  products  for  every  deliverable  specified  in  the  contract. 
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Schedules  must  be  compatible  with,  and  are  dependent  upon,  the  hardware  development 
schedule  for  which  the  technical  publication  are  being  prepared.  Planning  for  analysis, 
writer  and  illustrator  training,  and  writer  assignment,  must  be  compatible  with  information 
availability  which  in  turn  is  contingent  upon  the  basic  equipment  production  schedule. 
Scheduling  and  planning  carried  out  for  the  development  of  technical  publication  products 
must  reflect  this  compatibility  to  ensure  the  delivery  of  Tl  concurrent  with  the  hardware. 

3. 4. 2.5  BLOCK  NO.  3.2  - PERSONNEL  REQUIREMENTS  AND  ASSIGNMENTS 
Following  the  development  of  Tl  schedules,  the  management  and  administration  group 
should  identify  personnel  requirements  and  begin  to  make  personnel  assignments  in  accor- 
dance with  the  predicted  needs  of  the  program. 

3. 4. 2. 6 BLOCK  NO.  3.3  - FACILITIES  REQUIREMENTS  AND  ACQUISITION 
It  is  the  task  of  management  to  identify  facility  requirements,  to  develop  a realistic 
acquisition  schedule,  and  to  initiate  those  actions  necessary  to  acquire  the  needed 
facilities. 

3. 4. 2.7  BLOCK  NO . 3.4  - INTEGRATE  AND  COORDINATE  THE  ACTIVITIES  OF 
ALL  ELEMENTS  OF  THE  TECHNICAL  PUBLICATIONS  ORGANIZATION 

If  manpower  and  facilities  are  to  be  used  effectively,  their  utilization  must  be  coordin- 
ated and  integrated  not  only  across  organizational  elements,  but  also  within  elements. 

3. 4. 2. 8 BLOCK  NO.  3.5  - MONITOR  PERFORMANCE 

It  is  also  management's  responsibility  to  monitor  and  review  the  performance  of  the 
various  production  elements  of  the  technical  publications  organization  in  order  to 
insure  that  o quality  product  is  delivered  as  scheduled.  Management  must  have  well 
developed  means  and  mechanisms  for  monitoring  performance  and  intermediate  products 
in  order  to  insure  final  delivery. 
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3.4.2. 9 BLOCK  NO.  3.6  - ADJUST  SCHEDULE  AND  MAN^OADING  AS 
REQUIRED 

Since  it  is  unlikely  that  any  hardware  development  program  will  be  implemented  as 
planned,  the  technical  publications  organization  management  must  be  responsive  to 
the  changing  hardware  schedules  by  adjusting  man/loading  and  schedules  in  order  to 
avoid  inefficient  use  of  resources  and  high  periodic  work  loads.  If  the  hardware  system 
delivery  date  is  delayed  by  two  months,  the  data  acquisition  for  that  system  will  also 
be  delayed.  Management  most  either  reassign  personnel  or  pay  the  price  of  lost  man- 
hours. Lost  man-hours  early  in  the  program  can  only  lead  to  peak  loads  and  premium 
pay  os  the  delivery  dates  approach.  Constant  shifting  and  adjusting  of  personnel  and 
assignments  In  accordance  with  the  hardware  program  are  essential  if  wasted  man-hours 
and  premium  pay  are  to  be  avoided. 

3.4.2.10  BLOCK  NO.  3.7  - MEET  PRODUCT  DELIVERY  DATE 

The  major  goal  of  management  is  to  deliver  a quality  product  as  scheduled.  This  does 
not  imply  that  the  Initial  schedule  is  necessarily  the  one  to  be  met.  It  is  the  adjusted 
schedule  based  upon  hardware  development  that  Is  the  critical  date. 

3.4.2.11  BLOCK  NO.  4 - DATA  ACQUISITION  AND  DISTRIBUTION 

Data  are  the  raw  material  of  the  Technical  Information  Generation  process.  Without 
a formal  effective  data  acquisition  and  distribution  system,  effective  technical  pub- 
lications cannot  be  assured.  Data  acquisition  is  of  prime  importance  since  it  provides 
the  basts  for  the  publication  content.  The  process  should  be  formalized  and  systemitized; 
and  the  significance  reflected  in  the  technical  publications  organization. 

3.4.2.12  BLOCK  NO.  4.1  - PREPARE  DATA  ACQUISITION  SCHEDULE 
Data  requirements  for  each  technical  publication  product  should  be  prepared  and 
availability  dates  identified  on  a schedule  in  order  to  ensure  the  earliest  possible 
acquisition  of  each  and  every  required  datum. 
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3.4.2.13  BLOCK  NO.  4.2  - DEFINE  ACQUISITION  AND  DISTRIBUTION 
PROCEDURES 


Data  are  of  no  use  to  o writer  unless  they  are  available  to  him.  Thus,  formal  data- 
acquisition  and  data  distribution  procedures  should  be  established  by  the  data-acquisition 
group.  The  users  of  the  data  should  be  informed  and  instructed  in  the  standard  ac- 
quisition and  distribution  procedures. 

3.4.2.14  BLOCK  NO.  4.3  - DEFINE  FACILITY  REQUIREMENTS  AND  ACQUIRE 
In  the  event  that  planning  indicates  additional  facilities  are  required,  it  is  the  re- 
sponsibility of  this  group  to  take  the  necessary  action  to  acquire  the  needed  facilities. 

3.4.2.15  BLOCK  NO.  4.4 -ACQUIRE  DATA  AND  DISTRIBUTE 

Once  the  procedures  are  implemented  and  facilities  are  available  for  data  distribution, 
it  is  the  responsibility  of  this  group  to  both  acquire  and  distribute  the  data  as  scheduled. 
When  it  is  cost  effective,  this  group  may  notify  users  of  data  availability  and  location 
in  lieu  of  acquisition  and  distribution . 

3.4.2.16  BLOCK  NO.  4.5 -MAINTAIN  LOGS  AND  FILE 

It  is  the  responsibility  of  the  data  acquisition  and  distribution  group  to  maintain  logs 
of  the  physical  location  of  data  which  are  not  and  will  not  be  physically  co-locoted 
with  the  group.  Data  availability  and  physical  location  must  be  recorded  for  co-located 
ond  non  co-located  data. 

3.4.2.17  BLOCK  NO.  5 - ANALYSIS 

Analyses,  which  are  implemented  by  the  Navy  during  the  Tl  Definition  phase  of  NTIPS, 
should  be  provided  to  the  contractor  for  his  review  and  to  enable  the  contractor  to  identify 
additional  analysis  requirements. 

3.4.2.18  BLOCK  NO.  5.1  - REVIEW  ALL  PRELIMINARY  ANALYSES 

It  is  the  contractar's  responsibility  to  review  all  preliminary  analyses  Implemented  by 
the  Navy  during  the  Tl  Definition  phase  of  the  program. 
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3.4.2.19  BLOCK  NO.  5.2  - ITERATE  AND  UPDATE  ALL  ANALYSIS  AS  REQUIRED 
Since  ft  Is  the  contractor  who  develops  the  detailed  information  and  detailed  dato  needed 

to  carry  out  a complete  and  detailed  analysis  of  the  technical  publications  requirements, 
the  contractor  should  Iterate  and  updote  the  onalysis  carried  out  by  the  Navy  in  the  Tl 
Definition  phose  of  the  program.  Contractor  analyses  should  be  carried  to  the  task  level. 

3.4.2.20  BLOCK  NO.  5.3  - PREPARE  PRELIMINARY  WRITER/ILLUSTRATOR 

TRAINING  REQUIREMENTS 

One  of  the  by-products  of  the  analyses  should  be  the  identification  of  writer  and  illus- 
trator preliminary  training  requirements.  When  completed  in  preliminary  form,  these 
requirements  should  be  turned  over  to  the  training  group  for  development  and  finaliza- 
tion of  training  plans. 

3.4.2.21  BLOCK  NO.  5.4  - MODIFY  SPECIFICATION  AND  DID  REQUIREMENTS 

AS  INDICATED 

The  analysis  group  should  also  be  responsible  for  establishing  modified  specification  and 
DID  requirements  when  Indicated  by  the  results  of  the  analysis.  The  contract  must  of 
course  be  suitably  modified  as  required  to  formalize  these  changes. 

3.4.2.22  BLOCK  NO.  5.5  - REVIEW  TRAINING  PUNS 

The  analysts  responsible  for  preparing  the  preliminary  writer  and  illustrator  training  re- 
quirements should  review  the  final  training  plan  prepared  by  the  training  group  for  com- 
pliance with  the  requirement  described  in  Block  5.3  above. 

3.4.2.23  BLOCK  NO.  5.6  - ITERATE  ANALYSES  AS  REQUIRED 

Tl  analyses  should  be  repeated  In  response  to  engineering  changes  when  such  chonges 
are  approved. 

3.4.2.24  BLOCK  NO.  5.7  - REVISE  REQUIREMENTS  AS  NECESSARY 
When  hardware  changes  require  on  analysis  iteration,  the  potential  for  revision  to 
specifications  and  DID's  must  also  be  evaluated  and  further  contractual  changes  made 
when  warranted. 
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3.4.2.25  BLOCK  NO.  6 - TRAINING 

Since  training  is  the  primary  means  for  controlling  the  quality  of  writer  data  interpreta- 
tion and  writer  and  illustrator  synthesis,  it  is  impossible  to  place  too  much  emphasis  on 
this  element  of  the  Technical  Information  Generation  phase  of  NTIPS.  Training  is  the  pri- 
mary technique  for  achieving  a high  quality  final  product  in  terms  of  technical  content, 
completeness,  and  accuracy. 

3.4.2.26  BLOCK  NO.  6.1  - REVIEW  PRELIMINARY  TRAINING  REQUIREMENTS 
The  training  group  will  develop  detailed  training  requirements  for  writers  and  illustrators 
based  on  the  preliminary  training  requirements  resulting  from  analyses. 

3.4.2.27  BLOCK  NO.  6.2  - PREPARE  TRAINING  PLAN  AND  SCHEDULE 

A detailed  plan  and  schedule  should  be  prepared  which  are  compatible  with  the  hardware 
development  and  technical  publication  schedules.  The  schedules  developed  for  training 
must  be  compatible  with  product-need  dates. 

3.4.2.28  BLOCK  NO.  6.3  - PREPARE  CURRICULA  AND  CERTIFICATION 
PROCEDURES 

Following  development  of  the  training  plan  and  schedule  the  training  group  should 
develop  detailed  curricula  and  certification  procedures.  Writer  certification  is  a means 
of  controlling  the  quality  of  training  and  assessing  the  effectiveness  of  training.  Certi- 
fication is  construed  as  an  essential  ingredient  of  the  Technical  Information  Generation 
phase  of  NTIPS, 

3.4.2.29  BLOCK  NO.  6.4  - IMPLEMENT  TRAINING 
This  block  is  self-explanatory. 

3.4.2.30  BLOCK  NO.  6.5  - CERTIFY  WRITERS  AND  ILLUSTRATORS 
Following  the  completion  of  training,  writers  and  illustrators  will  be  certified  for  the 
particulor  tosk  to  which  they  are  aulgned.  Certification  of  writers  and  illustrators 
should  be  a formal  process.  Editors  will  be  certified  as  writers  for  eoch  area  of  editing 
responsibility. 
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3.4.2.31  BLOCK  NO.  6.6  - REVISE  TRAINING  AND  CERTIFICATION  AS 

REQUIRED 

Continuous  evaluation  of  writer  and  illustrator  certification  procedures  should  be  carried 
out  until  there  is  clear  evidence  that  both  the  training  and  certification  procedures  are 
effective  in  producing  a quality  product. 

3.4.2.32  BLOCK  NO.  6.7  - IMPLEMENT  ADDITIONAL  TRAINING  AS  REQUIRED 
The  evaluation  process  may  dictate  a revision  in  the  training  or  certification  procedures 
and  also  Identify  areas  where  additional  training  is  required.  When  such  areas  are 
identified,  additional  training  should  be  Implemented  by  the  contractor. 

3.4.2.33  BLOCK  NO.  7 - WRITER 

Writers,  along  with  the  illustrators  (See  Block  No.  8)  are  the  production  personnel  of  the 
Technical  Publications  program.  All  other  Tl  Generation  activities  support  this  element. 

3.4.2.34  BLOCK  NO.  7.1  - PREPARE  SCHEDULE  FOR  EACH  PRODUCT 
Each  writer  should  be  provided  a detailed  schedule  for  each  product  for  which  he  has 
responsibility.  This  schedule  should  include  the  writer's  training  schedule,  the  avail- 
ability of  analysis  results,  and  the  availability  of  other  data.  Each  writer  will  then  be 
fully  aware  of  the  material  inputs  required  for  his  effort  and  immediately  detect  missing 
materials  (data)  and  other  potential  problems. 

3.4.2.35  BLOCK  NO.  7.2  - TRAINING 

Each  writer  should  receive  training  (as  cited  in  Block  6.4)  in  organizational  procedures, 
data-acquisition  procedures,  the  availability  of  analyses  results,  the  detailed  formal  re- 
quirements af  his  product,  hardware  design,  and  hardware  availability  schedules.  Editors 
will  be  trained  as  writers  in  each  area  of  editing  responsibility. 

3.4.2.36  BLOCK  NO.  7.3  - CERTIFICATION 

The  procedure  for  certification  of  a writer  (as  cited  In  Block  6.5)  should  provide  evidence 
of  satisfactory  completion  of  the  training  program  as  evidence  of  his  obility  to  perform  his 
assigned  task.  Editors  will  be  certified  as  writers  in  each  area  of  editing  responsibility. 
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3.4.2.37  BLOCK  NO.  7.4  - ACQUIRE  DATA  AND  KEEP  LOG 

It  should  be  the  responsibility  of  each  individual  writer  to  keep  o log  of  his  data  require- 
ments ond  acquisitions  including  both  material  and  personnel  contacts.  The  log  should 
reflect  the  completion  of  intermediate  data  products,  draft  reviews,  and  draft  validation. 

3.4.2.38  BLOCK  NO.  7.5  - PREPARE  TEXT  AND  ILLUSTRATION  REQUIREMENTS 
It  is  the  writer's  task  to  prepare  text  and  to  identify  illustration  requirements. 

3.4.2.39  BLOCK  NO.  7.6  - PREPARE  TEXT 

The  writer  should  prepare  his  text  and  update  his  log  on  a weekly  basis. 

3.4.2.40  BLOCK  NO.  7.7  - ASSEMBLE  DRAFT  FOR  EDITING 

It  is  the  writer's  responsibility  to  assemble  text  and  illustrations  into  a draft  suitable 
for  editing. 

3. 4. 2.-41  BLOCK  NO.  7.8  - COORDINATE  VALIDATION  PROCESSES  AND 

SCHEDULES 

Validation  is  a QA  process  which  must  be  planned  and  scheduled  by  the  writer  (and 
illustrator)  or  he  must  at  least  assist  the  QA  personnel  in  validation  planning  and 
scheduling. 

3.4.2.42  BLOCK  NO.  7.9  - SUPPORT  VALIDATION 

When  the  writer  has  planned  and  scheduled  (or  assisted  the  QA  personnel  in  planning 
and  scheduling)  the  validation  process,  he  will  support  the  implementation  of  the  valida- 
tion process  as  indicated  in  the  validation  plan. 

3.4.2.43  BLOCK  NO.  7.10  - REVISE  AND  REVALIDATE  AS  REQUIRED 
Following  initial  validation,  the  draft  (both  text  and  illustrations)  will  be  revised  as 
required  and  revalidated  if  necessary. 


3.4.2.44  BLOCK  NO.  7.11  - SUBMIT  FINAL  DRAFT 

Following  the  post-validation  revision  of  the  draft,  the  final  draft  (including  illustrations) 
in  the  required  medium  will  be  submitted  to  the  Navy  for  verification. 

3.4.2.45  BLOCK  NO.  7.12  - SUPPORT  VERIFICATION  PLAN  PREPARATION 
The  contractor  will  prepare  a verification  plan  bosed  on  Navy  requirements  in  order  to 
assist  the  Navy  In  planning.  The  writer  will  support  the  effort  of  developing  the  verifi- 
cation plan. 

3.4.2.46  BLOCK  NO.  7.13  - SUPPORT  VERIFICATION  AS  REQUIRED 

The  Navy  may  require  support  during  the  verification  process.  It  will  be  the  writer's 
responsibility  to  support  the  Navy  as  required  during  verification. 

3.4.2.47  BLOCK  NO.  7.14  - REVISE  AS  REQUIRED 

Following  verification,  it  is  the  writer's  responsibility  to  revise  text  and  requirements 
for  illustrations  as  indicated  by  the  results  of  the  verification. 

3.4.2.48  BLOCK  NO.  8 - ILLUSTRATOR 

The  illustrator  is  the  other  "production"  person  in  the  Tl  Generation  process  along  with 
the  writer.  The  writer  takes  the  lead  in  preparing  the  text  and  assembling  the  completed 
draft,  and  the  illustrator  must  meet  the  writer's  requirements. 

3.4.2.49  BLOCK  NO.  8.1  - PREPARE  SCHEDULE  FOR  EACH  PRODUCT 

In  order  for  Illustrations  to  be  prepared  efficiently,  the  efforts  and  assignments  of  each 
illustrator  should  be  scheduled  In  order  to  be  compatible  with  the  overall  technical  pub- 
lications organization. 

3.4.2.50  BLOCK  NO.  8.2  - TRAINING 

Illustrators  should  receive  training  in  the  same  manner  and  in  the  same  areas  as  the 

e 

writers. 
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3.4.2.51  BLOCK  NO.  8.3  - CERTIFICATION 

Illustrators  should  be  certified  under  the  same  program  as  the  writers. 

3.4.2.52  BLOCK  NO.  8.4  - ACQUIRE  DATA  AS  REQUIRED 

In  the  case  of  the  Illustrators,  the  detailed  illustration  requirements  are  provided  by  the 
writers  (Block  7,5)  and  form  an  essential  part  of  the  illustrators'  data  needs.  The  same 
procedures  which  are  used  to  provide  the  raw  materials  for  writers  should  be  used  to 
provide  illustrators  with  their  data  needs, 

3.4.2.53  BLOCK  NO.  8.5  - PREPARE  ILLUSTRATIONS 
This  block  is  self-explanatory. 

3.4.2.54  BLOCK  NO.  8.6  - SUPPORT  VALIDATION 

Illustrators  as  well  as  writers  may  be  required  to  support  the  validation  program.  If 
validation  support  is  required,  it  should  be  scheduled. 

3.4.2.55  BLOCK  NO.  8.7  - REVISE  ILLUSTRATIONS  AS  REQUIRED 
Following  the  validation  of  the  technical  publication,  the  illustrator  will  revise  his 
illustrations  as  required. 

3.4.2.56  BLOCK  NO.  8.8  - SUPPORT  VERIFICATION 

Illustrators  as  well  as  writers  may  be  required  by  the  Navy  to  support  verification. 

3.4.2.57  BLOCK  NO.  8.9  - REVISE  AS  REQUIRED. 

Following  verification,  the  illustrator  will  revise  the  illustrations  as  Indicated  by  the 
results  of  the  verification. 
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SECTION  4 

QUALITY  ASSURANCE  REQUIREMENTS 


4.1  GENERAL  DISCUSSION 

The  Navy  has  experienced  difficulty  in  obtaining  high  quality  Technical  Infomation 
on  a timely  basis,  even  when  adequate  specifications  have  been  established.  This 
problem  arises  as  a consequence  of  applying  QA  techniques  too  late  in  the  weapon- 
system  acquisition  process.  It  has  been  traditional  to  apply  QA  to  Technical  Infor- 
mation (Tl)  development  products,  beginning  with  drafts  of  the  final  documents.  Re- 
distribution of  QA  efforts  is  needed,  beginning  with  audits  of  the  completeness  and 
quality  of  raw  data  upon  which  the  documents  will  be  based. 

The  current  Navy  QA  program  Is  unmanageable.  The  Navy  does  not  have  the  resources 
to  fully  audit  the  total  product  from  the  various  contractors.  Accordingly,  the  approach 
to  QA  expressed  by  this  report  de-emphasizes  the  current  Navy  and  contractor  QA  pro- 
cedures of  monitoring  documents  and  products,  and  shifts  a major  portion  of  the  effort 
to  auditihg  the  process  of  developing  Tl.  The  contractor  should  assure  the  Navy  that 
the  Tl  development  process  is  being  done  correctly,  and  as  a result,  that  a reasonable 
probability  exists  that  the  output  will  be  adequate.  Figure  4-1  diagrams  a proposed  three- 
tiered  process  with  emphasis  on  the  front  end.  Navy  QA  personnel  audit  the  contractor's 
QA  personnel,  who  in  turn,  monitor  the  Tl  development  generation  process.  Figure 
4-2  depicts  the  current  QA  program  whose  emphasis  Is  on  editing  documents,  with  the 
major  portion  of  the  effort  spent  on  validation  and  verification  of  the  product.  A hard- 
ware system  contractor  currently  provides  approximately  one  QA  man  for  twenty  writers. 
The  Navy  monitors  the  contractor  QA  program  with  about  the  same  ratio  of  personnel. 

The  method  suggested  In  this  report  does  not  increase  this  ratio  for  auditing  the  proposed 
QA  program,  but  rather  provides  for  more  effective  and  efficient  use  of  these  personnel. 

With  greater  emphasis  of  the  customer  and  contractor  QA  on  the  earlier  phases  of  the 
development  of  the  Tl,  indications  of  the  expected  quality  of  the  Tl  can  be  formulated 
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and  a reduced  sampling  rate  can  be  used  in  later  phases  with  greater  confidence.  This 
would  eliminate  the  peak  loading  of  personnel  at  the  later  production  phases  and  would 
provide  the  Navy,  as  well  as  the  contractor,  with  early  indications  of  production  prob- 
lems of  the  Tl. 

So  that  the  Navy  QA  personnel  do  not  become  overburdened  through  maintaining  cognizance 
of  the  contractor's  activities  during  the  early  phases  of  the  Tl  development  process,  the 
Navy  QA  personnel  will  rely  primarily  upon  the  QA  reports  and  records  supplied  and 
maintained  by  the  contractor’s  QA  personnel.  The  Navy  personnel  will  only  perform 
sample  inspections  to  assure  authenticity  of  the  contractor's  QA  reports  and  records. 

MIL-Q-9858,  Quality  Program  Requirements,^  must  be  an  explicit  requirement  imposed 
upon  the  contractor.  The  QA  program  and  organization  implemented  by  the  contractor 
must  conform  to  this  specification  in  providing  the  documentation  for  QA  reports  and 
records. 

Implementation  of  an  effective  and  economical  Tl  Quality  Assurance  program  is  depend- 
ent upon  two  factors:  (1)  communicating  what  is  needed  and  desired  by  the  Navy  to  the 
contractor  and  his  suppliers,  and  (2)  providing  appropriate  mechanisms  for  management 
and  control  of  the  program.  Most  Navy  contractors  and  suppliers  wish  to  perform  well 
and  to  maintain  good  relations  with  the  Navy.  Establishment  of  a clear  understanding 
by  all  parties  as  to  their  responsibilities  for  Tl  quality  and  the  methods  and  procedures 
to  be  employed  should  normally  be  sufficient.  Sometimes,  however,  disputes  might 
arise  between  Navy  and  its  contractors.  In  such  cases,  and  more  importantly,  to  fore- 
stall such  cases,  it  is  necessary  that  appropriate  and  effective  mechanisms  for  manage- 
ment, control,  and  enforcement  of  requirements  be  Incorporated  into  the  controct.  This 
is  especially  important  when  a new  concept  for  QA  is  being  introduced. 

4.2  QUALITY  ASSURANCE  PROGRAM  PLAN 

The  principal  vehicle  for  achieving  an  effective  and  economical  Tl  Quality  Assurance 
program  is  the  Tl  Quality  Assurance  Program  Plan.  This  plan,  based  upon  the  concepts 
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and  philosophy  of  MIL-Q-9858,  Quality  Program  Requirements,^  should  be  developed 
during  the  proposal  phase  by  the  contractor  in  accordance  with  detailed,  specific  in- 
structions of  a Data  Item  Description  (DID)  prepared  for  this  purpose.  This  DID  should 
be  incorporated  into  the  Request  for  Proposal  (RFP)  package.  Each  bidder  should  be 
instructed  to  submit  the  plan  with  the  proposal.  The  RFP  should  make  it  clear  that 
upon  contract  award,  the  plan,  as  modified  in  accordance  with  agreements  reached 
during  contract  negotiations,  is  to  be  incorporated  into  the  contract  and  is  to  become 
the  implementing  document  for  the  program.  This  may  be  achieved  effectively  by  pre- 
paring the  Program  Plan  in  performance  specification  format.  It  should  also  be  under- 
stood that  the  proposed  Tl  Quality  Assurance  program  is  subject  to  disapprovol  by  the  Navy 
whenever  the  contractor's  proposed  procedures  do  not  offer  o high  probability  that  they 
will  accomplish  their  objectives.  The  Navy,  at  its  option,  may  furnish  written  notice 
of  the  acceptance  or  disapproval  of  the  contractor's  proposed  Tl  Quality  Assurance 
Program  Plan. 

4.2.1  GENERAL  Tl  QUALITY  ASSURANCE  PROGRAM  REQUIREMENTS 

All  Tl  development  processes,  products,  and  services  under  the  contract,  whether  developed, 

produced,  or  performed  by  the  contractor  or  its  subcontractors,  suppliers,  or  consultants, 
should  be  controlled  at  all  points  necessary  to  ensure  conformance  to  contractual  require- 
ments, including  the  requirements  of  the  approved  Plan.  The  program  should  provide  for 
the  prevention  or  ready  detection  of  discrepancies  or  deficiencies  at  any  stage  of  the  Tl 
development  program  and  should  provide  for  timely  and  positive  corrective  action.  The 
contractor  should  be  required  to  make  objective  evidence  of  quality  confarmance  readily 
available  to  the  Navy  representative. 

The  authority  and  responsibility  of  those  in  charge  of  Tl  planning,  development,  pro- 
duction, testing,  and  inspection  of  quality  should  be  clearly  stated  in  the  Plan.  The 
program  should  include  effective  control  of  purchased  materials  and  subcontracted  work. 
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4.2.2  Tl  QUALITY  ASSURANCE  PROGRAM  MANAGEMENT 

Effective  management  for  Tl  Quality  Assurance  should  be  clearly  established  and  main- 
tained by  the  contractor.  Personnel  performing  Tl  Quality  Assurance  functions  should 
have  sufficient,  well-defined  responsibility  and  authority,  and  the  organizational  free- 
dom needed  to  identify  and  evaluate  Tl  quality  problems  and  to  initiate,  recommend, 
or  provide  solutions.  The  contractor's  Management  personnel  should  be  required  to 
review  the  status  and  adequacy  of  the  Tl  Quality  Assurance  program  on  a regular  basis. 

The  contractor  should  be  required  to  maintain  and  use  any  records  or  data  essential  to 
the  economical  and  effective  operation  of  the  Tl  Quality  Assurance  program.  These 
records  should  be  made  readily  available  for  review  by  the  Navy  Representative  and 
copies  of  individual  records  or  data  should  be  furnished  by  the  contractor  upon  request. 
Records  are  considered  one  of  the  principal  forms  of  objective  evidence  of  quality.  The 
Tl  Quality  Assurance  program  should  employ  methods  and  procedures  for  insuring  that 
records  are  complete,  accurate,  and  reliable.  Records  of  discrepancies,  deficiencies, 
and  corrective  actions  taken  or  planned  should  be  established  and  maintained  by  the 
contractor.  As  a minimum,  these  records  should  indicate  the  number  and  nature  of 
observations  made  by  the  contractor's  Tl  Quality  Assurance  personnel,  together  with  the 
number  and  types  of  deficiencies  found.  The  contractor  should  provide  detailed  record- 
keeping procedures  employing,  as  apprapriate,  available  QA  discrepancy-reporting 
systems  or  adaptations  thereof.  The  QA  program  should  provide  for  the  analysis  and 
use  of  these  records  as  a basis  for  management  action.  Summary  reports  of  these  records 
and  analyses  of  trends  in  Tl  development  processes,  or  performance  of  work  to  prevent 
further  occurrences  of  deficiencies,  should  be  submitted  periodically  to  the  Navy  in 
accordance  with  the  CDRL. 

The  contractor  should  be  made  responsible  for  QA  of  Tl  processes  and  products  acquired 
from  subcontractors  and  suppliers.  The  contractor  must  be  required  to  employ  appropriate 
measures  to  ensure  that  each  subcontractor  maintains  the  required  levels  of  quality.  The 
contractor's  Tl  Quality  Assurance  personnel  should  establish  and  maintain  surveillance 
aver  subcantractors  at  all  levels  to  insure  satisfactory  performance  of  QA  tasks 
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and  to  insure  required  levels  of  quality  in  the  Tl  processes  and  products.  The  contractor 
should  be  required  to  schedule  and  conduct  on-site  surveys,  when  necessary,  to  insure 
compliance  with  quality  program  requirements.  Criteria  for  survey  performance  should 
be  based  upon  criticality  or  complexity  of  items  of  Tl  being  procured,  known  problems 
or  difficulties,  and  quality  history. 

4.3  TWO  PHASE  PROGRAM 

The  QA  inspection  criteria  will  vary  with  the  Tl  material  and  products  being  inspected. 

In  the  initial  phases  of  development,  the  quality  of  a product  is  predicted  by  "indicators" 
or  perhaps  more  precisely,  the  indicators  provide  notice  of  impending  quality  or  schedule 
difficulties  of  a Tl  production.  The  second  phase  relies  upon  inspection  of  specific  qual- 
ity characteristics  of  the  intermediate  and  final  Tl  products.  Consequently,  the  QA 
program  consists  of  two  phases. 

A discussion  of  statistical  methods  of  sampling  to  ensure  the  quality  level  of  the  Tl  Is 
also  presented.  The  statistical  procedures  can  be  used  in  conjunction  with  both  phases 
and  both  types  of  inspection  criteria. 

4.3.1  QUALITY  ASSURANCE  INDICATORS 

The  QA  indicators  can  be  used  to  the  greatest  advantoje  during  the  early  phases  of  the 
Tl  development.  The  early  phases  produce  no  inspectable  product,  but  rather  involve 
various  activities  that  are  crucial  to  the  successful  development  of  the  Tl.  The  per- 
formance and  course  of  these  activities  are  tracked  by  the  QA  indicators.  Though 
these  indicators  are  essential  in  the  early  development  of  the  Tl,  they  should  also 
prove  of  value  in  tracking  the  progress  of  the  Tl  development  through  the  later  stages, 
where  additional  inspections  of  the  products  are  performed. 

Certain  administrative,  analysis,  data-acquisitlon,  and  data-utilization  tasks  must  be 
performed  so  that  accurate,  useful  and  timely  Tl  can  be  generated.  Assurance  that 
these  tasks  are  performed  on  a timely  basis  serves  as  an  early  indication  of  the  timeli- 
ness and  quality  of  the  final  Tl  product.  Thus,  the  tracking  and  recording  of  the 
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performance  of  these  tasks  provide  the  quality  indicators.  Contractor  QA  personnel 
viill  not  be  required  to  monitor  all  the  Tl  development  through  each  step,  but  will  use 
sampling  techniques  to  insure  at  a sufficient  confidence  level  that  each  Tl  development 
is  progressing  at  a high  quality  level.  As  indicated  in  earlier  discussions.  Navy  QA 
personnel  will  then  sample  the  contractor  QA  records  to  ensure  compliance  to  the  system. 

No  new  processes  for  the  Tl  development  are  suggested  by  the  incorporation  of  the  QA 
indicators.  However,  additional  forms  and  record-keeping  will  be  required  by  the 
technical  writer.  This  additional  record-keeping  need  not  be  a burden  to  the  writer. 

On  the  contrary,  proper  formulation  of  the  record  forms  will  provide  the  writer  a useful 
guide  and  schedule  for  his  development  of  the  Tl.  Further,  the  formalization  of  a dato- 
management  program  will  establish  a ready  means  for  the  writer  to  obtain  required  data 
and  will  decrease  time  the  writer  must  spend  in  search  of  the  data.  QA  personnel  re- 
quirements will  be  increased  in  the  early  phases  relative  to  current  requirements  during 
these  phases;  however,  the  QA  requirements  in  the  later  phases  should  be  reduced  with 
a possible  overall  net  decrease  in  personnel  requirements. 

QA  personnel  will  be  expected  primarily  to  determine  whether  the  administrative, 
analytical,  data-acquisition,  and  data-utilization  tasks  have  been  performed  and 
whether  the  performance  of  these  tasks  has  been  within  schedule.  The  QA  personnel 
will  not  be  expected  to  be  able  to  judge  the  accuracy  or  quality  of  the  results  of  these 
tasks.  There  are  no  measurable  characteristics  to  determine  such  accuracy  or  quality. 

For  example,  in  the  data-acquisition  task,  the  quality  of  the  data  must  be  assumed  to 
be  accurate  if  the  material  represents  the  latest  release,  and  consequently  the  only 
criterion  for  inspection  is  whether  the  data  were  obtained  within  schedule  and  whether 
the  data  are  current  and  complete.  Further,  it  is  not  the  responsibility  of  the  QA  per- 
sonnel to  ascertain  the  accuracy  of  analysis  results,  (e.g.,  from  a User-Data  Match  or 
a Head/Book  Trade-off),  but  rather  to  insure  that  the  results  from  such  analyses  are 
based  on  the  most  recent  revision  of  available  data  and  are  complete  within  the  sched- 
uled time  period. 
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4.3.2  PRODUCT  INSPECTION 

The  product  inspections  ore  the  same  inspections  that  are  currently  performed  on  the 
Tl  products.  These  inspections  on  current  QA  functions  were  described  briefly  in  the 
previous  Section  2.1  (Block  5)  and  are  applicable  to  the  intermediate  and  final  products 
of  the  Tl  development. 

These  inspections  include  assessment  of  the  quality  of  the  mechanical  production  of 
the  Tl  and  of  the  composition  of  the  Tl.  Validation/verification  procedures  ensure  the 
Technical  accuracy  and  completeness  of  the  product.  The  validation  and  verification 
also  ensure  that  the  procedural  activities  described  in  the  Tl  are  accurate  and  efficient 
for  the  performance  of  the  task  and  that  the  Tl  is  applicable.  Revisions,  corrections, 
and  additions  or  deletions  to  the  Tl  can  result  from  the  validation  and  verification 
procedures. 

The  composition  inspections  are  concerned  with  spelling,  grammar,  technical  control, 
format,  logical  organization  of  data,  text  completeness,  continuity,  and  ease  of  access 
to  the  needed  information.  Inspections  will  also  be  implemented  to  insure  the  inclusion 
of  engineering  review  comments,  validation  corrections,  and  verification  revisions. 

The  production  quality  criteria  are  concerned  with  format  of  title  page  and  front  matter, 
figure  references  and  titles,  type  size,  figure  quality,  proper  page  makeup  and  number- 
ing, classification,  and  specification  compatibility. 

Contractors  have  extensive  checklists  with  inspection  criteria  for  these  characteristics 
of  the  Tl.  The  statistical  sampling  procedures  used  for  QA  indicators  may  also  be  applied 
to  the  inspections  of  these  characteristics.  To  inspect  oM^  Tl  for  the  possible  char- 
acteristics of  the  above  listed  categories  would  be  a limitless  task.  While  a large  part 
of  the  current  QA  effort  is  concerned  with  these  inspections,  it  is  considered  that  with 
a greater  use  of  sampling  techniques,  along  with  the  incorporation  of  the  process  of  using 
quality  indicators,  a more  prudent  and  efficient  inspection  of  the  Tl  products  will  be 
possible. 
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4.3.3  SAMPLING  (Data  Base  Quality  - Estimation) 


Regardless  of  the  ability  of  the  writer,  missing  or  obsolete  data  will  prevent  his 

writing  a high  quality  technical  document.  The  quality  of  the  data  base  he  uses 

can  be  evaluated  by  the  tedious  and  costly  100%  audit  method  (in  practice  even 

this  expenditure  of  time  and  effort  cannot  guarantee  that  all  errors  will  be  deleted). 

In  lieu  of  a 100%  audit  of  the  data  base,  statistical  sampling  can  be  used  to  estimate 

the  quality  of  the  data  base.  The  overall  practicality  of  using  sampling  to  estimate 

quality  is  illustrated  by  increasing  usage  by  defense  agencies  such  as  the  Defense 

5 

Contract  Audit  Agency  (DCAA).  For  additional  rationale  for  using  sampling  see 
MIL-STD-  105  D,  Sampling  Procedures  and  Tables  for  Inspection  by  Attributes.  ^ 

In  the  paragraphs  that  follow,  a statistical  test  plan  is  outlined  which  is  tailored  to 
estimate  the  quality  of  the  Tl  development  and  products.  The  test  plan  is  applicable 
to  both  the  QA  indicators  and  the  product  inspections. 

4.3.3. 1 STATISTICAL  PLAN 

The  Tl  data  base  is  assumed  to  contain  information  for  k systems  with  n.  data  items 

th  ^ 

for  the  k system.  "System"  is  used  as  a generic  term  in  this  development,  it  can 

refer  to  a hardware  system,  a subsystem,  or  any  other  desired  classification  of  hardware 

items.  The  data  items  will  be  coded  as  a "0"  (zera)  or  a "1"  (one)  in  accordance  with 

the  following: 

"1"  = data  item  is  fully  acceptable 

"0"  = data  item  is  not  fully  acceptable;  i.e.,  it  is  absent  or  late 
A multiple  stage  sampling  plan  is  outlined  in  Figure  4-3, 
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Sfep  3 


Note:  If  required  sample  aize  is  fractional,  round  to  the  next  larger  integer, 


FIGURE  4-3 


STEP  1:  Select  a 15%  sample  of  the  data  items  or  3 data  items  (whichever  is  larger)  at 
random  from  system  k (e.g.,  system  1).  If  there  are  no  data  items  classified  as 
"0"  skip  the  remaining  steps  and  go  on  to  System  k + 1 (e.g.,  system  2).  If  there 
is  one  data  item  classified  as  "0"  ~ proceed  to  Step  2.  If  there  are  2 or  more 
data  items  classified  as  "0"  skip  Step  2 and  proceed  to  Step  3. 

STEP  2:  Given  that  one  data  item  was  classified  as  "0"  in  Step  1,  then  select  a random 
sample  of  on  additional  35%  of  the  data  items  in  System  k , If  there  are  no 
additional  data  items  classified  as  "0"  skip  Step  3 and  go  on  to  system  k + 1. 

If  there  ore  one  or  more  additional  data  items  classified  as  "0",  proceed  to 


STEP  3:  Given  that  one  or  more  data  items  in  Step  2 were  classified  as  "0"  or  if 

two  or  more  data  items  in  Step  1 were  classified  "0",  then  conduct  a 100% 
audit  of  the  remaining  data  items  in  System  k,  ~ then  go  on  to  System 
k + 1 . 

STEP  4:  Correct  all  data  items  classified  as  "0"  's. 


When  all  systems  have  been  sampled,  summarize  the  results  in  the  following  format: 


System  ID 

Number  Data  Items 
Inspected 

Number  Data  Items 
Classified  "1"  (one) 

Number  Data  Items 
Classified  "0"  (zero) 

1 

2 

3 

• 

• 

• 

k 

TOTALS 

Methods  to  estimate  the  quality  of  the  dota  base  from  these  tabulated  results  will  be 
presented  in  the  next  section, 

* 

The  analysis  will  be  based  on  the  assumption  that  data  item  classification  is  a Bemouilli 
random  variable.  The  only  assumptions  required  are  that  data  items  can  be  classified  as 
acceptable  or  unacceptable,  and  that  a randomly  selected  data  item  will  have  a prob- 
ability of  being  acceptable,  P,  which  is  between  zero  and  one  inclusive.  This  random 
variable,  X,  Is  characterized  by  its  probability  function  as  follows: 

Prob  (X  = 1)  = P for  0 < P < 1 

Prob  (X  = 0)  = 1 - P 


* classified  as  "0"  or  "1 
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The  mean  of  this  distribution  is  given  by; 


1 • Prob  (X  = 1)  + 0 • Prob  (X  = 0)  = P (2) 

The  variance  of  this  distribution  is  given  by; 

• Prob  (X  = 1)  + 02  • Prob  (X  = 0)  - (Mean)^  = P - P^  (3) 

(Note  that  the  variance  increases  as  P drops  from  1 to  .5,  achieving  its  maximum 
at  P = .5.) 

The  probability  that  a randomly  selected  data  item  from  system  k wilj  be  classified  "1" 
has  probability  Pj^  . Pj^  is  an  unknown  parameter  which  can  be  estimated  from  the 
results  of  a random  sample. 

Let  denote  the  number  of  items  in  the  sample,  chosen  from  system  k,  which  were 

classified  as  "1 

Let  denote  the  number  of  items  in  the  sample,  chosen  from  system  k,  which  were 
classified  as  "0". 


An  unbiased  estimator,  Pj^,  of  the  unknown  parameter  Pj^  is; 


A 


P|^  is  statistically  known  as  a point  estimator  of  Pj^,  . It  may  be  larger  than  Pj^  or  smaller 
than  P,  . Management  decisions  are  frequently  made  to  guard  against  "worst  case" 
quality  problems.  An  estimated  100  •%  confidence  lower  bound  on  Pj^  can  be  computed 
based  on  the  sample  data.  The  calculations  are  more  complex,  but  they  have  been 
extensively  discussed  and  tabulated  in  NAVWEPS  Report  No.  8090,  "Binomiol  Reliability 
Toble  (Lower  Confidence  Limits  for  the  Binomial  Distribution)".^  Denote  this  lower  bound 
by  P|^  lovyer*  ^ confidence  level,  100  V%,  is  associated  with  this  lower  bound  to  specify 
how  sure  one  wants  to  be  that  the  estimated  lower  bound  is  lower  than  Pj^.  (This  procedure 
is  the  same  as  computing  a 100V%  one-sided  confidence  interval  on  Pj^. 


- gamma 
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Both  the  point  estimate  and  the  lower  bound  or  worst  case  estimator  should  be  com- 
puted for  each  system.  If  repeated  sampling  Is  carried  out  about  50%  of  the  estimators 
will  exceed  but  only  100  (1  - V)%  of  P|^  will  exceed  Pj^. 

An  overall  measure  of  the  quality  of  the  data  base  will  be  formuloted  in  a flexible 
manner  to  allow  the  Navy  the  opportunity  to  weight  some  systems  more  heavily  than 
others. 


Defin.  = W,  P,  + Wj  Pj  +....+  P,^ 

where  the  Pj^  are  previously  defined, 

where  the  are  weights  such  that 

1.  W|^  21  0 

2.  2.,  W,  = 1 = All  k 

k k 

P ^ 

tot  can  be  estimated  using  the  previous  estimates  of  Pj^  's  as  follows: 

L = W,  P,  + Wj  P^  + . . . . 


The  Navy  determines  the  weights  based  on  the  relative  importance  of  the  systems.  If 
the  Novy  does  not  wish  to  make  a judgement  about  the  relative  weights,  then  set 
= 1 for  all  k. 


A value  for  can  be  used  to  pass  or  fail  the  Tl  data  base  (it  is  a measure  of  overall 

quality  ranging  from  "0"  for  extremely  poor  quality  to  "1"  for  a very  high  quolity  data 
bose).  The  approach  described  provides  the  evaluator  of  a Tl  data  base  with  measures 
of  the  average  quality  and  "worst  case"  quality  for  each  system  and  with  a measure  of  the 
overall  quality  of  the  Tl  data  base.  A summary  of  the  statistical  parameters  and  estimators 
used  to  measure  quality  is  presented  in  the  following  chart. 


STATISTICAL  PARAMETERS  AND  ESTIMATORS 


Symbol 

Description 

How  Obtained 

Percentage  of  data  items  in  system  k 

^ unknown 

that  are  fully  acceptable 

(P|^  used  as  estimator) 

A 

Unbiased  estimator  of  P, 

k 

computed  from 
data 

k lower 

Lower  bound  on  Pj^  such  that 

computed  from 

lower’  = ” 

where  100^%  is  the  confidence  level 

data 

Weighted  percentage  of  data  items  across 

unknown 

all  systems  that  are  fully  acceptable 

A 

(Pj^^^  used  as  estimator) 

A 

Unbiased  estimator  of  P.  , 

tot 

computed  from 
data 

4.3.3.2  SAMPLING  PIAN  CHARACTERISTICS 

The  sampling  plan  previously  presented  has  been  designed  to  be  practical  and  is  based 
on  fundamental  statistical  principles.  Practicality  demands  that  the  plan  be  simple 
to  keep  sampling  costs  to  a minimum  and  to  maximize  the  intuitive  understanding  of  the 
plan.  A basic  statistical  principle  is  to  sample  more  frequently  from  systems  (categories) 
that  have  larger  variances  than  from  systems  that  have  smaller  variances  (see  Des  Raj, 
Sampling  Theory  ®).  To  apply  this  principle  to  the  present  situation  v/e  noted  that 
systems  with  decreasing  quality  (decreasing  probability  that  a sampled  data  item  would 
be  classified  "1")  had  larger  variances  (see  para,  4.3.3. 1),  Therefore,  systems  indicat- 
ing lower  quality  should  be  sampled  more  frequently,  which  is  also  intuitively  appealing. 
The  proposed  plan  radically  increases  the  sample  size  in  a system  upon  indication  of 
quality  problems. 


T'le  sensitivity  of  the  plon  can  be  assessed  quantitatively  by  computing  the  probability 
of  detecting  unacceptable  data  items  in  the  data  base  as  a function  of  the  frequency  of 
existence  of  unacceptable  data  items  in  the  data  base.  As  the  quality  level  In  a system 
decreases,  the  probability  of  detecting  the  lack  of  quality  Increases  rapidly.  The  sam- 
pling procedure  has  most  difficulty  detecting  Isolated  data-Item  deficiencies,  but  if 
some  problem  (such  as  schedule  slippage  causing  late  releases  of  data  Items  for  o system) 
Is  systematically  generating  deficient  data  Items,  then  the  plan  would  have  good  sensi- 
tivity. Three  examples  follow  which  show  the  sensitivity  (probability  of  detecting  a "0") 
of  the  plan  for  various  numbers  of  defective  Items  in  the  system. 

Example  1:  A system  has  twenty  (20)  dota  Items  from  which  a sample  of 
size  3 is  to  be  selected. 

Number  of  the  20  data  items  Probability  of  getting  I or 

that  are  "0"  's  more  "0”  's  In  sample 


2 

.284 

4 

.509 

6 

.681 

8 

.807 

10 

.895 

12 

.951 

14 

.982 

16 

.996 

18 

1.000 

20 

1.000 

Example  two  shows  that  the  sensitivity  of  the  first  stage  increases  with  increasing 
sample  sire  even  if  the  percentage  of  defective  Items  remains  constant  (compare  with 
example  I), 
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Example  2:  A syrtem  has  forty  (40)  data  items  from  which  a sample  of  size  6 
(15%)  is  to  be  selected. 


Number  of  the  40  data 
items  that  are  ”0"  's 

Probability  of  getting 
more  "0"  's  in  sample 

4 

.493 

8 

.764 

12 

.902 

16 

.965 

20 

.990 

24 

.998 

28 

.9998 

32 

.999993 

36 

1.000 

40 

1.000 

Example  three  shows  the  sensitivity  of  the  second  stage  of  the  sampling  plan  to  the 
number  of  deficient  items  in  the  system.  In  order  to  be  in  stage  2^  some  indication 
of  a quality  problem  was  exhibited  in  stage  1;  therefore,  it  is  desirable  to  have 
much  greater  sensitivity  in  the  second  stage.  Comparison  with  example  1 shows 
this  to  be  true. 

Example  3:  A system  has  twenty  (20)  data  items.  During  the  first  stage,  an 
item  classified  as  a "0"  appeared  in  the  sample  of  size  3.  An 
additional  sample  of  size  7 is  taken. 

Number  of  the  remaining  seventeen  (17)  Probability  of  getting  1 or 

data  items  that  are  "0"  '$  more  "0"  '$  in  2nd  stage  sample 


0 

2 

4 


0 

.669 

.912 
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(Example  3 - Table  - Confinued) 


6 

.983 

8 

.998 

10 

.99995 

12 

1.000 

U 

1.000 

16 

1.000 

17 

1.000 

All  calculations  in  examples  are  based  on  Hypergeometric  Probability  Distribution  since 
sample  replentishment  is  not  assumed. 


Informal  Note:  Hypeigeometric  Probabilities 


Notation:  Let 


{n  = Total  items  in  a category  n = 1,  2,  3,  ... 
s = Number  of  the  n items  sampled  o 1 s - n 

t = Total  "bad"  items  of  the  n o^  t 5n 

r = Number  of  "bad"  items  in  the  sample  of 
size  s o<ri$in,  oirit 


The  probability  of  exactly  r "bad"  items  in  the  sample  is: 

(:)(;::) 

:) 

( Hypergeometric  ) 

o otherwise 

. t t (n  - t)  I s!  (n  - s)  ! 

rl(t-r)  ! (s-r)!  (n-t-$+r)!  n! 
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Therefore,  the  probability  of  1 or  more  "bod"  items  beingsampled  is: 


1 - Prob  ( 0 "bod"  items  in  sample  ) = 


1 - (n  - t)  I (n  - s)  ! 
(n  “ t - $)  I n I 


(i.e.,  when  n = 0) 


4.4  IDENTIFICATION  OF  QUALITY  ASSURANCE  INDICATORS 
A discussion  with  examples  of  QA  indicators  for  each  phase  of  Tl  development  is  pro- 
vided in  the  following  sections.  As  previously  discussed,  the  concept  of  indicators 
was  developed  primarily  to  serve  the  early  phases  of  Tl  Generation.  However,  it  may 
in  part  apply  to  other  phases  as  well. 


4.4.1  TECHNICAL  INFORMATION  GENERATION 

This  is  the  phase  in  which  the  QA  indicators  ore  used.  In  the  early  stages  of  this  phase, 
critical  tasks  are  performed  that  will  insure  timely  Tl  with  quality.  Thus,  this  phase  is 
discussed  in  detail  with  examples  for  each  stage.  Figure  4-4  is  a Sample  Form  for  list- 
ing the  relevont  QA  indicators.  This  figure  will  provide  a standard  method  for  showing 
the  use  of  QA  indicators  in  auditing  the  development  of  this  program. 

4.4. 1.1  INDICATORS  FOR  AUDIT  OF  THE  DATA  MANAGEMENT  PROGRAM 
DEVELOPMENT 

The  organizotion  and  establishment  of  the  Data  Management  Program  represent  the 
cornerstone  of  performing  QA  audits  of  the  activities  during  the  early  stages  of  Tl 
development.  This  program  establishes  the  data-gathering  network  procedures  and  the 
data-logging  procedures.  Therefore,  the  monitoring  of  the  development  of  the  Data 
Management  Progrom  by  QA  personnel  is  of  considerable  importance.  The  format  of 
Figure  4-4  will  be  used  for  listing  the  relevant  QA  Indicators  presented  in  Table  4-1. 

4.4. 1.2  INDICATORS  FOR  AUDIT  OF  THE  DESIGN  ANALYSIS 

The  design  analysis  consists  of  all  the  analyses  required  for  formulating  the  content(s) 
and  form  of  the  Tl.  The  format  selection,  User-Data  Match,  and  Head/Book  Trade-Off 
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QUALITY  INDICATOR  RECORD 


DATA  MANAGEMENT  PROGRAM  DEVELOPMENT  AUDIT 


Quality  Indicotors 

1 . Tl  Data  Organization  Developed 

2.  Data-Logging  Procedures  Developed 

3.  Data -Cataloging  System  Developed 

4.  DatO'Storage  Capacity  Acquired 

5.  Data-Logging  Forms  Developed 

6.  Management  Forms  Developed 

7.  Liaison  with  Dato-Release  Authority  Established 

8.  Procedure  for  ECP  Routing  to  Tl  Group  Established 

9.  Release-Schedule  Tracking  Formalized 

10.  Liaison  with  Design  and  Functional  Groups  Established  and  Formalized 


TABLE  4 - 1 


are  updated  and  cost  analyses  for  the  Tl  are  performed.  The  results  of  these  analyses 
become  vital  data  for  the  production  of  the  Tl.  The  QA  personnel  cannot  ascertain  the 
accuracy  of  the  results  of  these  analyses,  just  as  QA  personnel  cannot  insure  the  accuracy 
of  a stress  analysis  for  the  design  of  hardware.  The  completeness  and  the  timeliness  of 
the  results,  however,  can  be  audited  by  QA  personnel.  A list  of  sample  QA  indicators 
for  this  stage  of  the  Tl  development  is  shown  on  Table  4-2.  The  format  of  Figure  4-4 
should  be  followed  for  recording  the  QA  record  of  the  Design  Analysis  Program. 

4. 4. 1.3  INDICATORS  FOR  AUDITING  THE  DATA  ACQUISITION  PROCESS 

The  data  acquired  for  the  Tl  development  provides  the  basis  for  the  information  contained 
in  the  Tl.  Acquisition  of  such  data  must  be  complete  and  on  schedule  to  insure  that  the 
Tl  is  accurate  and  complete.  The  data  con  take  many  forms,  such  as  engineering  reports 
and  drawings,  technical  orders  for  Government-furnished  equipment,  vendor  brochures, 
and  data  and  reports  from  the  various  support  disciplines.  Data  can  also  be  obtained 
through  personal  communications  with  other  personnel  knowledgeable  of  the  system  and 
by  attending  design  reviews.  A sample  list  of  the  quality  indicators  for  data  acquisition 
is  shown  in  Table  4-3.  In  many  cases  there  will  be  more  than  one  quality  indicator  for 
a given  data  item,  (i.e.,  ECP's,  LSAR's,  MEAR's,  Hazard  Analyses,  SER's,  PPB's),  so 
space  should  be  provided  for  individual  listings.  Table  4-4  lists  the  quality  indicators 
involved  in  data  acquisition  and  the  development  of  the  personnel  contacts  with  personnel 
in  other  disciplines.  The  recording  format  for  these  quality  indicators  should  be  similar 
to  the  format  of  Figure  4-4.  A "Document  Entered  and  Available  Date"  column  and  a 
"Location"  column  should  be  included  with  the  "Scheduled  Release  Date"  and  "Actual 
Release  Date"  columns. 

4. 4. 1.4  INDICATORS  FOR  AUDITING  THE  DATA  UTILIZATION  PROCESS 
Data  utilization  is  the  general  title  given  to  writer  activities  and  training  depicted  in 
Blocks  6,  7,  and  8 of  Figure  3-3.  Data  utilization  audits  are  made  to  determine  if  the 
writer  uses  the  appropriate  information  that  is  available.  In  this  stage,  the  writer  is 
selected  and  assigned  the  development  of  a given  piece  of  Tl  or  a group  of  Tl  items. 
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DESIGN  ANALYSIS  AUDIT 


Quality  Indicators 

1.  Data  Accessing/lndexing  Procedure  Developed 

2.  Format  Selection  Analysis  Update  Completed 

3.  Format/Medium  Selected 

4.  User-Data  Match  Analysis  Update  Complete 

• Definition  of  User  Characteristics  Complete 

• Tl  Presentation  Format  Selected 

5.  Head/Book  Trade-Off  Analysis  Update  Complete 

• Tl  Required  Data  Specified 

6.  Tl  Specifications  and  Data  Item  Descriptions  Modified 


TABLE  4-2 
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DATA  ACQUISITION  LOG  AUDIT 


Quality  Indicators 

1.  Procurement  and  Test  Specifications 

2.  Proposed  Technical  Description 

3.  Photos  of  Mockups  or  Equipment 

4.  Support-Equipment  Description  Data 

5.  Task  Analysis  Data 

6.  Maintenance  Concept  as  Applicable 

7.  Special  User  Personnel  Qualifications 

8.  Notes  and  Materials  from  Vendors 

9.  Failure  Modes  and  Effects-Analysis  Data 

10.  Engineering  Reports 

11.  Blueprints/Drawings/Sketches 

12.  Vendor  Brochures  and  Commercial  Manuals 

13.  Engineering  Change  Proposals  (ECP's) 

14.  Logistics  Support  Analysis  Report  (LSAR's) 

15.  Maintenance  Engineering  Analysis  Records  (MEAR's) 

16.  Hazard  Analysis 

17.  Subsystem  Hazard  Analyses 

18.  Support  Equipment  Requirements  Sheets  (SER's) 

19.  Provisioning  Parts  Breakdowns  (PPB's) 

20.  Development  Hardware  Location  and  Availability 

TABLE  4-3 
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DATA  ACQUISITION  TASK  AUDIT 


Quality  Indicators 

Task  I - documentation 

1.  Release  Schedules  Established 

• Engineering  Drawings 

• Engineering  Descriptions 

• Specifications 

• Task  Analysis 

• Maintenance  Engineering  Analysis  Data  (MEA) 

• Hazard  Analysis 

• Support  Equipment  Requirement  Sheets  (SER's) 

2,  Engineering  Design  Review 

Task  II  - TECHNICAL  PERSONNEL  LIAISON 

1 . Design  Engineer 

2.  Maintenance  Engineer 

3.  Training  Engineers 

4.  Reliability  Engineer 

5.  Human  Factors  Engineer 

6.  System  Safety  Engineer 

7.  Production  Supervisor 

TABLE  4-4 
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The  writer  also  undergoes  training  and  certification  during  the  data  utilization  phase. 
The  certification  process  ensures  that  the  writer  is  knowledgeable  of  the  specifications 
and  requirements  imposed  by  the  customer,  understands  the  analyses  results,  and  is 
knowledgeable  of  the  company's  special  Tl  formats  and  guides  for  writers.  The  writer  is 
also  trained  in  the  organization  and  logging  of  data  and  in  the  overall  data-monogement 
program.  Table  4-5  lists  the  quality  indicators  which  could  be  used  by  QA  personnel  to 
track  and  audit  the  performance  of  data  utilization  functions. 

4.4. 1.5  INDICATORS  FOR  DRAFT  PREPARATION  AUDIT 

The  QA  indicators  as  shown  in  Table  4-6  follow  the  progress  of  the  preparation  of  the 
Tl  draft.  No  effort  is  made  by  the  QA  personnel  to  edit  the  draft,  but  only  to  monitor 
its  progress.  The  recording  format  for  this  table  should  be  similar  to  Figure  4-4. 

4. 4. 1.6  INDICATORS  FOR  DRAFT  VALIDATION  AUDIT 

Validation  of  the  draft  Tl  is  performed  by  contractor  personnel  at  the  contractor's 
facility.  The  QA  indicators  shown  in  Table  4-7  are  concerned  with  the  preparation 
for  completion  of  validation  and  the  insertion  of  corrections  (resulting  from  the  valida- 
tion) Into  the  draft.  Records  should  be  in  a format  similar  to  Figure  4-4,  with  a Draft 
Validation  Audit  form  for  each  piece  of  Tl  to  be  validated.  It  is  expected  thot  the 
validation  will  be  based  on  the  real  hardware,  and  will  not  merely  consist  of  a visual 
inspection. 

4.4.2  INDICATORS  FOR  CARRYING  OUT  THE  Tl  FINAL  REVIEW 
The  later  phases  of  the  Tl  preparation  ore  grouped  together  for  discussion  because  these 
are  the  phases  where  definite  products  are  involved  and  inspections  become  the  prime 
QA  method,  rather  than  use  the  quality  Indicators.  However,  QA  indicators  are  still 
a useful  means  of  insuring  proper  progress  of  the  Tl  development  during  these  phases, 
and  examples  of  the  indicators  are  provided  in  Table  4-8.  Records  should  olso  be  in  a 
format  similar  toFigure  4-4.  Again,  the  indicators  measure  the  scheduled  completion 
of  the  tasks  to  be  performed  in  these  phases. 
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DATA  UTILIZATION  AUDIT 
Quolity  Indicators 

PART  I - WRITER  QUALIFICATIONS 

1.  Wril’er's  Name 

2.  Wrlfer  Certification 

3.  Writer  Selection 


PART  II  - DATA  ITEM  CHECKOUT  LOG 
1 . Data  Item 

(Each  item  of  data  will  be  listed,  with  dates  for  logging  in  and  out) 

PART  III  - WRITER  LIAISON  RECORD 
1 . Group  or  Person  Contacted 

(Each  group  or  person  contacted  will  be  listed  with  date  of  contact) 


part  IV  - NAVY  QA  PERSONNEL  REVIEW  MEETING 
1 . NAVY  QA  Contoct 

(Navy  QA  personnel  contacted  will  be  listed  and  date  of  contact) 


TABLE  4-5 
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DRAFT  PREPARATION  AUDIT 


Quality  Indicotors 

1.  Tl  Outline  Complete 

2.  Illustration  Requirements  to  Illustrators 

3.  Illustrations  Complete 

4.  Draft  Complete 

• 20%  Complete 

• 40%  Complete 

• 60%  Complete 

• 80%  Complete 

5.  Draft  Edited 

6.  Draft  Proofed 


TABLE  4-6 


DRAFT  VALIDATION  AUDIT 


Quality  Indicators 

1.  Validation  Plan  Complete 

2.  Hardware  Available 

3.  Support  Equipment  Available 

4.  Personnel  Notified  and  Available 

5.  Validation  Performed 

6.  Draft  Revisions  Formulated 

• Text 

• Illustration 

7.  Draft  Revisions  Incorporated 


TABLE  4-7 
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Tl  FINAL  REVIEW 


Quolity  Indicotors 

1.  Typographic  Composition 

2 . Proofing  of  Text 

3.  Reproducible  Text 

4.  Produce  Customer  Review  Copies 

5.  Review  Package  to  Navy 

6.  Incarporate  Navy  Comments 

7.  Final  Navy  Review  and  Appraval 

8.  Release  for  Preliminary  Replication 

9.  Preliminary  Distribution  to  Navy  of  Final  Copies  for  Verification 

10.  Preparation  of  Navy  Verification  Plan  and  Support  of  Navy  Verification 

11.  Receive  Verification  Comments 

12.  Corrections  to  Tl  Masters 

13.  Produce  Navy  Review  Copies 

14.  Review  Package  to  Navy 

15.  Navy  Review  and  Approval  of  Update 

16.  Release  Update  for  Publication 

17.  Distribution  of  Update  to  Navy 
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4.5  DRAFT  INSPECTION  PROCEDURES  AND  CRITERIA 

The  draft  is  an  Intermediate  product  and  Is  inspectable.  The  draft  can  be  edited  within 
a technical  writing  group  to  the  same  criteria  as  the  QA  personnel  use  in  inspecting 
the  Tl  draft.  The  QA  personnel  may  use  the  same  sampling  techniques  in  the  draft  in- 
spections as  are  used  for  the  QA  indicators.  The  draft  inspections  will  generally  cover 
format,  composition,  and  content.  The  technical  accuracy,  completeness,  and  useful- 
ness are  determined  during  validation  of  the  draft.  The  contractor  QA  will  perform 
sampling  inspections  and  Navy  Quality  Assurance  personnel  will  sample  the  contractor 
QA  records  to  ensure  adherence  to  the  quality  plan.  This  is  the  same  multi-tiered  QA 
procedure  discussed  previously. 

The  criteria  for  the  inspections  are  well  documented  and  In  current  use.  Each  con- 
tractor hos  checklists  of  criteria  for  Tl  draft  inspections.  Some  variations  among  these 
lists  exist,  but  all  are  composed  of  criteria  set  forth  by  the  various  Tl  specifications. 

A list  showing  examples  of  the  existing  Tl  specifications  is  given  in  Table  4-9.  There 
are  general  specifications  for  Tl,  and  specifications  for  Tl  for  different  types  of  weapons 
systems,  equipment,  and  tasks,  as  illustrated  by  the  table.  There  are  also  different 
specifications  for  different  Tl  presentation  formats  such  as  checklists,  illustrations,  and 
procedures. 

The  contractor  criteria  checklists  are  derived  from  specifications  which  are  applicable 
to  the  particular  Tl  draft.  A list  of  criteria  is  given  by  Harold  E.  Price  in  a report  en- 
titled "Development  of  a Draft  Specification  for  Technical  Manual  Quality  Assurance. " 
Price's  list  of  criteria  is  similar  to,  but  more  comprehensive  than,  checklists  derived 
by  contractors.  Table  4-iO  is  a modification  of  Price's  list.  It  is  o sompling  of  three 
criteria  of  acceptability  from  each  of  21  topics  taken  directly  from  Price.  Only  63 
criteria  are  listed  in  the  Table  from  a list  of  260  covering  over  twenty  pages  in  the 
referenced  report.  The  entire  process  of  QA  may  include  such  complex  inspection  tests 
as  readability  and  factor  determination. 
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SAMPLE  LISTING  OF  Tl  SPECIFICATIONS  ** 


MIL-M-38784 

MIL-M-38789A 

Ml  L-M-38793 
MIL-C-8122b(AS) 
MIL-1 -28947 

MIL-M-I5017G 

MIL-M-22202B 

MIL-M-23305 

MIL-M-23618D 

MIL-M-23619A 


MIL-M-7298C 

MIL-M-7700A 

MIL-M-8910 

MIL-M-9854B 

MiL-M-81218 


Manuals,  Technical;  General  Requirements  for  Preparation  of 

Manual,  Technical,  Overhaul  Instructions  with  Illustrated 
Parts  Breakdown  (For  Various  Equipment  and  Accessories) 

Manual,  Technical,  Calibration  Procedures,  Preparation  of 

Checklists;  Flight  Crew,  Preparation  of 

Illustrations  for  Technical  Repair  Parts  Publications, 

Preparation  of 

Manuals,  Technical:  Equipments  and  Systems  Content  Require- 
ments for 

Manual,  Technical,  Cross  Servicing,  Schedule  (Aircraft) 
Preparation  of 

Manual,  Technicol  Operation,  Maintenance  and  Overhaul 
Instructions  with  Illustrated  Parts  Breakdown  (For  Aircraft 
Launching  and  Recovery  Equipment)  Preparation  of 

Manual,  Technical,  Periodic  Maintenance  Requirements, 
Preparation  of 

Manual,  Technical,  Periodic  Maintenance  Requirements, 
Maintenance  Requirements  Cards,  Sequence  Control  Charts, 

And  Related  Test  Flight  Check  List,  for  Air  Weapons  Systems, 
Printing  and  Binding  of 

Manuals,  Technical:  Commercial  Equipment 

Manuals,  Flight 

Manuals,  Technical;  Illustrated  Parts  Breakdown;  Preparation  of 

Manual,  Technical,  Structural  Repair  (For  Aircraft) 

Manual,  Technical,  Aircraft  Engine,  Intermediate  and  Depot 
Maintenance,  Preparation  of 


**  Note:  This  listing  denotes  various  types  of  specifications  for  Tl,  but  not 
necessarily  those  reviewed  for  this  report. 
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TABLE  4-10 


EXAMPLE  DRAFT-INSPECTION  PARAMETERS 
AND  CRITERIA  OF  ACCEPTABILITY  * 


1.  ORGANIZATION 

A.  Principol  Units  and  Work  Packages 

1 . Manual  organizaf’ion  is  based  on  a hierarchy  of  maintainable  units  or 
equipment. 

2.  Manual  organization  is  compatible  with  a family  tree  or  topdown 
breakdown  of  maintainable  units  or  equipment. 

3.  Presentation  of  equipment  or  work  units  follows  a logical  organization  such  as 
order  of  use,  maintenance  hierarchy,  or  is  arranged  according  to  data  flow. 

B.  Arrangement  within  Sections  or  Pockoges 

All  information  concerning  a maintainable  unit  is  consolidated  into  a 
single  package. 

2.  The  content,  format,  and  sequence  of  information  within  a package 
are  standardized. 

3.  A "local"  table  of  contents  is  included  to  indicate  material  available 
within  a section. 

C.  Composition  Practices 

there  is  approximately  one  heading  for  every  two  paragraphs. 

2.  Headings  are  set  off  by  spacing,  printed  in  bold  lettering,  or  otherwise 
emphasized. 

3.  The  ratio  of  major  headings  to  subheadings  is  not  larger  than  1 to  10. 

D.  Prose-Grophic  Balance 

TT  Approximately  half  of  the  material  in  the  manual  is  text  and  half  is 
graphic. 

2.  Any  imbalance  between  narrative  and  pictorial  favors  pictorial  in  order 
to  minimize  procedural  errors  and  performance  time. 

3.  Pictorial  information  is  provided  when  steps  refer  to: 

a.  a particular  equipment  location, 

b.  the  relationship  between  two  or  more  equipment  items, 

c.  a specific  equipment  manipulation  (e.g.,  valve  rotation), 

d.  o test  readout  which  is  continuous  (e.g.,  waveform)  rather  than 
discrete  (e.g.,  numerical  value). 

2.  PROSE  COMPOSITION 

A.  General  Style  Principles 

T!  A nomenclature  pictorial  is  presented  at  the  beginning  of  a section. 

2.  Pictorials  are  used  to  introduce  key  equipment  terminology  used  in 
prose  discussions. 

3,  Short  familiar  names  are  used  for  hardware  once  it  has  been  introduced. 

For  example,  after  referring  to  a "videodisc  proportional  spacing  word 
processor  with  high  speed  printer, " further  discussion  can  simply  use  the 
"word  processor", 
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B,  Instnjctional  Writing 

1 . N inety  percent  of  the  maintenance  instructions  are  proceduralized. 

2.  Each  procedural  step  has  na  more  than  three  sentences. 

3.  "Second  person  imperative"  is  used  for  instructions,  e.g.,  "Turn 
the  Sensitivity  Control  to  maximum", 

C,  Paragraphs 

1 , Short  thought  units  of  "chunks"  of  information  are  used  in  preference 
to  longer  units. 

2,  Paragraphs  are  short  and  have  tapic  sentences, 

3,  Longer  sections  of  material  start  with  a summary. 

D,  Sentences 

1 . Sentences  generally  contain  a single  idea  or  thought. 

2.  Sentences  are  limited  to  17  to  20  vrords. 

3.  Common  and  unambiguous  verbs  are  used. 

E,  Words 

TTTRere  is  an  average  of  1 .5  syllables  per  word  based  on  a count  of 
at  least  100  words. 

2,  Lengthy  words  are  eliminated  or  "diluted".  Use  common  words  in 
place  of  less  familiar  words. 

Simple 
about 
after 
be  couse 
call  for 
enough 
find  out 
for 
get 
if 

now 
of 

since 
use 

3.  Simple  prepositions  are  used. 

Simple 
like 
to 

by,  under 
to 

about 
so  that 

A-'iA 


Awkward 
along  the  lines  of 
in  order  to 
in  accordance  with 
with  a view  to 
with  reference  to 
with  the  result  that 


Awkward 

as  to,  with  regard  to 
subsequent  to 

inasmuch  as,  for  the  reason  that 
necessitate,  require 
sufficient 

ascertain,  determine 
for  the  purpose  of 
acquire,  obtain,  receive 
in  the  event  that 
at  this  time,  presently 
relative  to,  with  regard  to 
inasmuch  as 
utilize 
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F,  Non-Text  Words  and  Phrases 

1 . Labels  and  titles  of’  graphics  indicate  what  they  are  about. 

2.  Nomenclature  in  the  text  is  defined  the  first  time  it  is  used. 

3.  Headings  are  no  more  than  3 or  4 words. 

G,  Legibility 

1 . Page  layout  generally  enhances  reading  speed  and  comprehension. 

2.  Double  column  format  is  used  for  prose. 

3.  Ideal  line  length  is  between  14  picas  (2-5/16")  and  23  picas  (3-13/16"). 
One  pica  equals  0.166  incles. 

3.  GRAPHICS  COMPREHENSION 

A,  General  Graphics  Principles 

T!  General  appearance  ol''  each  graphic  page  is  good.  There  are  no 
smudges,  white  space  is  evenly  distributed  and  graphics  appear 
uncrowded. 

2.  Relation  of  elements  on  a page  is  clear. 

3.  Graphic  fidelity  is  kept  to  the  minimum  needed  for  the  task  at  hand. 

B,  Graphic  Form  Selection 

1.  "Symptom-probable  cause- remedy"  troubleshooting  charts  are  used 
where  possible  causes  do  not  number  more  than  13  items,  and  where  not 
more  than  two  or  three  of  these  are  probable. 

2.  System-analysis  checklists  are  used  in  complicated  checkout  procedures 
which  require  checkout  of  more  than  one  component. 

3.  Photographs  are  used  instead  of  drawings  in  cases  where  drawings 
would  need  more  than  very  common  geometrical  shapes  to  represent 
the  components. 

C,  Schematic  and  Wiring  Diogram  Practices 

1 . Piece  part  detail  is  only  shown  when  replacement  is  authorised  at  that 
level . 

2.  The  superordinate  units  containing  the  circuitry  ore  identified. 

3.  Procedural  support  is  provided  for  complex  portions  of  wiring  and 
schematic  diagrams. 

D,  Network  Diagram  Practices 

1 . '  Different  information  elements  on  diagrams  are  separated  by  distance, 

shading,  boxes,  or  some  other  method. 

2.  The  information  elements  of  network  diagrams  are  either  familiar 
symbols,  readily  understandable  alphanumerics  or  labels,  or  pictorials. 

3.  MDCs  (Maintenance  Dependency  Charts)  connectives  and  leader  lines 
do  not  overlap  function  codes. 
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E,  Block  Diagram  Practices 

1 . Direction  arrows  ore  used  to  show  flow. 

2.  Flow  is  left  to  right  and  top  to  bottom. 

3.  Height  to  width  ratio  of  2 to  3 is  used  for  functional  flow  blocks. 

F,  Illustrations  Practices 

TI  Line  drawings  are  used  to  emphasize  relevant  items  rather  than  to  faithfully 
portray  physical  fidelity. 

2.  Relation  of  parts  in  exploded  views  is  indicated  by  axis  lines  or  other 
obvious  connecting  lines. 

3.  Any  feature  of  a drawing  referred  to  in  the  text  is  at  least  as  large 
as  the  text  type  size. 

G.  Freestanding  or  Series  Pictorials 

TI  In  a series  of  drawings,  the  first  one  provides  orientation  with  respect 
to  the  major  end  item  of  equipment. 

2.  Arrows  or  lines  show  progression  from  less  to  more  detailed  drawings. 

3.  Relation  of  elements  in  a series  of  pages  of  sequential  figures  is  clear. 

H.  Table  Practices 

TT  Interpolation  is  not  required  in  numerical  look-up  tables. 

2.  Title  reflects  content  variables. 

3.  Headings  are  clear  and  scale  units  are  included  as  needed  in  headings. 

I.  Graph  Practices 

TI  Graphs  usually  have  only  a single  concept. 

2.  Comparison  between  data  is  shown  on  the  same  graph. 

3.  Graph  scales  do  not  distort  the  intended  relationships. 

J,  Photograph  Proctices 

TI  Phot^raphs  show^ equipment  from  the  technician's  view. 

2.  Photographic  views  indicate  uncommon  shape  or  dangerous  components 

3.  Photographs  are  retouched  to  help  orientation  of  the  workers. 


Not  all  factors  inspected  are  of  equal  importance  or  criticality  to  the  quality  of  the  Tl. 

A judicious  ranking  of  the  criticality  of  each  factor  within  o given  category  might  fur- 
ther enhance  the  quality  of  the  Tl  by  providing  a criticality  bias  to  the  sampling  per- 
formed. Editors  and  inspectors  currently  performing  100%  inspections  of  all  factors 
ore  perhaps  already  unconsciously  biased  in  this  manner.  Experimentation  might  prove 
biased  sampling  to  be  a viable  alternate  to  a straight  random  sampling  of  the  material 
to  be  inspected. 

The  validation  of  the  draft  determines  the  accuracy,  completeness,  and  usability  of  the 
material.  The  validation  process  determines  whether  the  Tl  is  correct  and  whether  the 
Tl  Is  sufficient  to  allow  the  desired  task  to  be  performed  efficiently,  safely,  and  without 
error.  Price's  report  (referenced  above)  provides  a checklist  for  determining  the  Job 
Performance  Adequacy  of  a Tl.  A sample  of  these  parameters  and  criteria  is  provided 
in  Table  4-11 . This  Table  provides  the  proper  task-performance  emphasis  that  is  appli- 
cable to  the  validation  phase  of  the  Tl  Generation.  Table  4-11  specifically  addresses 
the  validation  (or  verification)  process  and  suggests  a procedure  to  follow  in  dciermin- 
ing  the  quality  of  the  product. 

The  validation  of  the  Tl  is  an  explicitly  stated  contractual  requirement  and  documentary 
evidence  must  be  supplied  by  the  contractor  to  show  that  the  procedure  was  performed. 
Further,  errors  and  discrepancies  in  the  Tl  found  by  validation  are  documented  and  lists 
of  these  errors  are  provided  to  the  Navy  together  with  appropriate  corrective  actions. 

4.6  DISCUSSION  OF  PStUDO  OR  NON-NTIPS  Tl  QUALITY 
If  the  user  of  the  technical  information  has  difficulty  in  locating  and  interpreting  infor- 
mation contained  in  manuals,  the  utility  of  the  data  is  correspondingly  reduced.  Because 
of  the  mass  of  technical  information  required  to  support  a weapon  system,  some  mechanism 
internal  to  the  Navy  must  be  developed  to  provide  the  technical  publications  organization 
close  contact  with  user  personnel.  This  close  contact  should  serve  two  purposes:  1)  to 
ensure  the  availability  of  the  dota  and  to  aid  the  user  in  identifying  accessing  techniques 
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TABLE  4-n 


EXAMPLES  OF  VALIDATrON  PARAMETERS  AND  CRITERIA 
OF  ACCEPTABILITY  FOR  TECHNICAL  MANUALS  * 

1.  ACCESS  AND  SEARCH 

A.  Identificotion 

1 . The  Htle  page  reveals  the  purpose  of  the  manual  and  the  system  or 
equipment  for  which  it  is  used. 

2.  A full-page  Illustration  of  the  equipment  is  provided  near  the  front 
of  the  manual. 

3.  Manual  breakdown  is  based  on  a hierarchy  of  maintainable  units 
or  work  packages. 

B.  Sections  or  Packages 

1 . A "local " table  of  contents  is  used  to  indicate  material  available 
within  a section  or  package. 

2.  Test  equipment  is  listed  in  tables  to  show  how  and  when  it  is  needed. 

3.  Tables  show  availability  of  test  points  for  use  in  trouble-shooting 
procedures. 

C.  Table  of  Contents  and  Headings 

1 . The  table  of  contents  has  a logical  sequence  from  topic  to  topic. 

2.  The  table  of  contents  consistently  lists  all  text  headings  of  the  same 
level . 

3.  Each  heading  and  subheading  is  referenced  at  least  once  in  the  table 
of  contents  or  index . 

D.  Index 

1.  Any  text  heading  not  in  the  table  of  contents  is  in  the  index. 

2.  The  minimum  number  of  index  headings  Is  one  per  page  of  manual 
content . 

3.  The  index  is  set  up  in  alphabetical  order. 

E.  Reference  Compliance 

TT  Official  part  number  and  nomenclature  used  on  title  page  to  identify 
the  technical  manual.  (MIL-M-38784) 

2.  Inclusion  of  listing  of  technical  directives  (changes  and  bulletins) 
covered  by  the  manual . 

3.  Applicable  safety  precautions  included  .(MlL-M-38784) 

2.  USABILITY  AND  ACCEPTANCE 

A,  Information  Content  Adequacy 

TT  Initial  system  conditions  for  maintenance  activities  are  specifically 
identified  for  such  factors  as: 

a.  gaining  access 

b.  removing  fluids 

* This  table  taken  from  Reference  9 by  H.  E.  Price 
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c.  pressurizing/depressurizing 

d.  raising/lowering 

e.  energizing/de-energizing 
f . warm  up/cool  down 

2.  Initial  equipment  conditions,  prior  to  operations  or  maintenance,  are 
specifically  identified  for  such  factors  as; 

a,  switch  position 

b,  valve  position 

c,  recorder  settings 

d,  meter  settings. 

3.  Maintenance  checkpoints  are  identified  and  discussed  in  terms  of: 

a.  their  location 

b.  appropriate  measurement  equipment 

c.  tolerances  at  each  checkpoint 

d.  parameters  for  alternate  modes 
e . unusual  procedures  or  cautions 

B.  Job  Relevance  and  Efficiency 

All  information  concerning  a maintainable  unit  is  consolidated  into 
a single  package. 

2.  Manual  sections  or  packages  contain  only  that  information  required 
to  support  job  performance. 

3.  Information  not  relevant  to  the  job  or  excess  detail  is  eliminated. 

C.  Workpgce  and  User  Compatibility 

the  manual  specifies  wtiat  might  be  done  to  make  the  working  environment 
as  nearly  ideal  as  possible. 

2,  Specific  instructions  are  given  for  special  maintenance  which  might 
be  required  in  unusual  climatic  conditions  such  as  cold,  heat,  wind, 
altitude,  and  noise. 

3.  The  manual  is  constructed  so  as  to  allow  the  removal  and  addition  of 
pages. 

D.  Technical  Scope  and  Accuracy 

tecnnical  scope  and  accuracy  are  not  amenable  to  a checklist  type  of 
review.  Tests  for  scope  and  accuracy  are  appropriately  a part  of 
Validation  and  Verification  rather  than  In-P/ocvss  Reviews.  The  questions 
of  concern  are: 

1.  Is  the  information  in  the  manual  technically  accurate? 

2.  Is  there  enough  information  in  the  manual  to  adequately  support- 
job  performance? 

As  part  of  the  In-Process  Review,  however,  a form  of  adequacy  check  may 
be  performed.  NAVAIR  00-25-600  defines  an  adequacy  check  and  suggests 
a procedure  as  indicated  below; 
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The  objective  of  on  adequacy  check  is  to  determine  the  degree  to 
which  depth  and  scope  of  coverage  in  maintenance  manuals,  and 
the  parts  listing  and  breakdown  in  IPB's  (Illustrated  Parts  Breakdowns) 
ore  sufficient  to  support  repairobles,  replaceables,  and  items  to  be 
assembled  or  manufactured  within  the  Navy  estaL'ishment. 

To  perform  on  adequacy  check,  it  is  advantageous  to  work  in  groups, 
one  person  working  with  the  Provisioning  Parts  Bit  jkdown  (PPB),  one 
with  the  IPB  and  one  with  the  maintenance  manual.  The  person  with 
the  PPB  searches  out  items  coded  repairable,  replaceable,  to  be  assem- 
bled, or  to  be  manufactured.  The  part  number  and  nomenclature  of  these 
items  ore  called  out  along  with  the  source  code,  and  the  person  working 
with  the  IPB  determines  if  it  contains  the  proper  artwork  and  parts  break- 
down. The  person  working  the  maintenance  manual  ascertains  if  the 
depth  of  coverage  supports  the  source/maintainability/recovery  code. 

At  the  same  time,  the  approved  support-equipment  listing  can  be  checked 
to  insure  that  support  equipment  called  out  in  the  maintenance  procedure 
is  approved  and  compatible. 


which  minimize  fhe  time  required  to  identify  the  needed  technical  documentation, 
and  2)  to  provide  for  a feedback  loop  for  correcting  any  deficiencies  found  in  the 
technical  documentation  by  the  user.  In  order  to  facilitate  the  corrections  of  defi- 
ciencies, the  normal  chain  of  command  should  be  bypassed  to  permit  direct  access  to 
the  organization  responsible  for  making  the  correction. 
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SECTION  5 

RECOMMENDATIONS  AND  CONCLUSIONS 


5.1  GENERAL 

The  review  of  the  documentation  on  current  technical  publications  development  in 
NTIPS  has  led  to  the  general  conclusion  that  all  of  the  contractor  activities  should 
be  formalized  and  defined  carefully  by  the  contractor  prior  to  the  initiation  of  any 
Tl  program.  There  are  formal  requirements  for  text  generation  and  illustration  prepar- 
ation in  the  form  of  specifications  and  handbooks.  There  are  no  formal  requirements 
for  the  processes  of  data  acquisition,  analysis  of  technical  publication  requirements 
and  writer  training.  In  order  for  Tl  Generation  to  be  economical,  efficient,  and  of 
high  quality,  careful  planning  must  be  carried  out  in  the  areas  of  data  acquisition, 
analysis,  and  writer  and  illustrator  training.  In  order  to  ensure  contractor  compliance 
with  the  requirements,  the  Navy  must  develop  procurement  documentation  in  the  form 
of  Specifications,  Standards,  and  Data  Item  Descriptions.  Close  surveillance  of  the 
contractor  Tl  Generation  program  by  contractor  QA  personnel  is  also  necessary. 

5.2  PROGRAM  PLAN 

As  part  of  his  proposal,  the  contractor  should  prepare  a detailed  Program  Plan,  prefer- 
ably in  specification  format,  to  permit  Navy  evaluation  of  the  contractor's  understand- 
ing of  the  overall  Technical  Publications  effort.  Prior  to  contract  award,  the  Navy 
should  review  and  request  changes  as  necessary  to  the  plan  submitted  by  the  contractor, 
and  permit  only  approved  changes  to  the  implementation  of  that  plan  during  the  contract 
performance. 

5.3  DATA  MANAGEMENT 

The  data  management  process  is  crucial  to  the  success  of  Tl  Generation.  The  effective- 
ness of  the  Tl  is  in  direct  proportion  to  the  adequacy  and  availability  of  the  data  used 
as  the  raw  material  for  Tl  Generation.  It  Is  therefore  strongly  recommended  that  the 
data  management  process  be  formalized  and  monitored  by  both  management  and  QA 
personnel . 
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5.4  ANALYSES  REPLICATION 

In  order  for  the  analyses  to  be  useful,  they  must  be  carried  out  to  the  task  level  during 
the  contractor  replication  of  the  Navy  preliminary  analyses.  However,  no  analysis 
should  be  undertaken  unless  the  products  of  the  analysis  in  both  content  and  format  are 
directly  usable  by  the  writer.  To  ensure  that  the  results  of  the  anolysis  are  available 
when  needed  by  the  writer,  analyses  planning  and  scheduling  should  be  formalized  and 
addressed  in  the  Program  Plan. 

5.5  WRITER,  ILLUSTRATOR,  EDITOR,  TRAINING  AND  CERTIFICATION 
Training  is  the  primary  mechanism  for  controlling  writer,  editor,  and  illustrator  be- 
havior. It  is  essential  therefore,  that  this  training  be  comprehensive  and  complete, 

not  only  with  respect  to  the  technical  nature  of  the  task,  but  also  in  the  area  of  organi- 
zational procedures  which  will  facilitate  the  successful  implementation  of  the  writing 
task.  The  formal  development  of  effective  comprehensive  training  programs  should  be 
monitored  closely  by  both  management  and  QA  personnel. 

5.6  DATA  UTILIZATION 

Records  of  the  data  used  by  writers  and  illustrators  should  be  maintained  in  order  to 
verify  that  the  writer  has  available  the  most  recent  information  to  serve  as  the  raw 
material  for  his  final  product.  The  writer  should  be  responsible  for  nxiintaining  a 
record  of  this  material. 

5.7  DRAFT  REVIEW  AND  FEEDBACK 

An  essential  mechanism  for  shaping  writer  behavior  is  continuous  evaluation  of  his 
intermediate  data  products.  Mechanical  aspects  of  the  review  should  be  implemented 
with  word-processing  techniques  and  should  include  vocabulary  and  syntactical  elements 
which  may  be  programmed.  This  method  of  objective  review  of  the  writer's  intermediate 
products  eliminates  many  mechanical  and  Iterative  editing  tasks  and  permits  the  editor 
to  review  more  complex  and  subjective  aspects  of  text  and  illustration  preparation. 

When  editors  are  relieved  of  the  minor  mechanical  tasks,  they  may  implement  the 
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subjective  review  more  quickly  and  thus  shorten  the  feedback  time  cycle  for  writers  and 
illustrators.  The  Navy  Quality  Assurance  personnel  should  ensure  that  effective  con- 
tractor Quality  Assurance  programs  are  in  turn  making  sure  that  the  process  for  evalua- 
tion and  feedback  is  functioning  well. 

5.8  DRAFT  VALIDATION 

Draft  validation  should  be  procedural,  and  based  on  actual  hardware.  Visual  valida- 
tion and  validation  by  inspection  of  the  document  are  both  unacceptable  techniques. 
Validation  provides  the  contractor  with  a first  test  of  the  usability  of  his  product.  It 
must  be  recognized  that  a well-written,  syntactically  correct  document  may  contain 
technical  errors  which  may  not  be  discovered  until  an  attempt  is  made  to  use  the  docu- 
ment in  the  manner  for  which  it  is  intended.  Care  should  be  taken  in  the  selection  of 
maintenance  and  operating  personnel  who  will  implement  the  task  during  the  validation 
process.  Whenever  possible,  these  personnel  should  be  representative  of  the  user  popu- 
lation . 

5.9  SCHEDULING,  DATA  RELEASES  AND  SOURCES,  ECP's,  MODIFICATIONS 
AND  REVISIONS,  PRODUCTION  RELEASES  AND  SCHEDULES 

The  significance  of  scheduling  all  of  the  tasks  associated  with  Tl  Generation  cannot  be 
overemphasized.  Scheduling  is  the  key  to  ensuring  technically  correct  Tl.  Tl  Genera- 
tion schedules  should  be  based  upon  hardware  development  and  configuration  control 
programs.  Each  data  release,  ECP,  specification  modification,  and  production  schedule, 
for  each  piece  of  equipment  for  which  technical  publications  are  being  prepared,  should 
be  monitored  and  introduced  into  the  Tl  Generation  program  in  order  to  maintain  input- 
data  control  during  the  development  of  the  technical-publication  product.  Only  through 
this  procedure  can  accurate,  current,  and  complete  technical  information  be  included 
into  the  final  product. 

5.10  CONCLUSION 

While  it  is  suggested  that  all  of  the  above  recommendations  be  incorporated  into  NTIPS, 
each  recommendation  is  independent  of  the  other  and  may  be  implemented  individually. 
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APPENDIX  “A" 
GLOSSARY  OF  TERMS 


ACCURACY  - The  degree  of  conformity  to  a standard  or  model;  also  degree  of 
freedom  from  error  and  discrepancies. 

ADEQUACY  - Depth  and  scope  of  coverage  relative  to  a depth  and  scope  which 
is  sufficient  to  support  all  tasks  and  functions  at  the  prescribed  level  of 
the  user  consistent  with  the  equipment  to  be  used. 

AUDIT  - A selective  comparison 
standards  or  objectives. 


CUSTOMER  - Navy  activity  responsible  for  the  QA  effort  to  ensure  adequacy 
and  accuracy  of  the  technical  manual. 


draft/draft  text/  manuscript  - The  technicol  data  prepared  in  relaxed 

format  to  allow  use  of  cost  effective  practices  and  techniques.  The  prep- 
aration and  depth  of  coverage  of  the  data  must  be  prepared  in  general 
accordance  with  the  applicable  technical  manual  specification. 

FEEDBACK  - Information  provided  to  a person  performing  a task  which  describes 
the  effectiveness  of  the  performance. 

FLESCH  COUNT  - A measure  of  reading  grade  level  that  involves  determination 
of  the  average  number  of  sentences  per  paragraph,  the  average  number  of 
words  per  sentence,  and  the  average  number  of  syllables  per  word. 


HEAD/BOOK  TRADE-OFF  - An  analysis  to  determine  how  much  technical  infor- 
mation should  be  included  in  manuals  and  job  aids,  and  how  much  technical 
information  should  be  included  in  training  curricula.  These  categories 
are  not  intended  to  be  mutually  exclusive. 
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INSPECTIONS  - The  examination  and  testing  of  supplies  and  services  to  determine 
whether  they  conform  to  specified  requirements. 

MANUSCRIPT  - (see  Draft) 

MEDIUM  - The  substratum  upon  which  the  information  is  "written". 

QUALITY  ASSURANCE  - A planned  and  systematic  pattern  of  all  actions  necessary 
to  provide  adequate  confidence  that  the  item  or  product  conforms  to  estab- 
lished technical  requirements. 

QUALITY  ASSURANCE  INDICATOR  - An  observable  quality  program  event. 

REPLICATION  - The  process  of  reproducing  technical  information  in  the  form  and 
in  the  medium  required  by  the  contract. 

SYNTAX  - Orderly  arrangement  of  words  as  elements  in  a sentence  (sentence  structure). 

TECHNICAL  INFORMATION  (Tl)  - All  types  and  forms  of  technical  publications 
(partial  and  complete)  procured  by  the  Navy  with  hardware  systems  far  the 
purposes  of  maintenance,  operation,  training,  and  logistic  support. 

Tl  DEFINITION  - This  NTIPS*  phase  is  the  initial  phase  of  the  program.  It  includes 
data  collection;  planning;  and  CDRL,  DID's,  TMCR's,  and  detailed  Tl  specifi- 
cations; and  analyses  necessary  for  the  preparation  of  the  Request  for  Proposal 

(RFP). 

Tl  GENERATION  - This  NTIPS*  phase  includes  data  collection  and  management, 
preparing  technical-publications  planning  documents,  writing  the  technical 
manual,  critiquing  the  technical  manual,  and  performing  Validation. 

USER(S)  - Those  personnel  (i.e.,  operators,  instructors,  maintainers,  technicians) 
wl  must  apply  the  technical  publication  in  its  actual  use. 

* NTIPS  - Navy  Technical  Infom.ution  Presentation  System 
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USER-DATA  MATCH  - An  analysis  to  determine  precisely  what  technical  infor-  j 

mation  the  user  requires  to  perform  an  assigned  task  in  the  work  environ*  j 

ment  in  accordance  with  user  characteristics  and  capabilities. 

VALIDATION  - The  process  by  which  the  contractor  tests  a manual  for  technical 
adequacy  and  accuracy.  This  is  accomplished  by  actual  performance  of 
the  operating  and  maintenance  instructions  (procedural  validation)  on 
the  system/equipment  for  which  the  technical  manual  is  written. 

VERIFICATION  - The  process  by  which  a technical  manual  is  tested  and  proved 

by  the  using  activity,  under  procuring  activity  jurisdiction,  to  be  adequate 
for  operation  and  maintenance . 
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